2024-03-25T13:59:30.864 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T13:59:30.864 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266 --verbose --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/orig.config.yaml 2024-03-25T13:59:30.911 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T13:59:30.998 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621266' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8uAacY53DXOKDITja8/1iob9sZhkTWetwQMGMhL6oONG+hzEmsexaheGDGSltZaL6A1lH72N2TmBFw8y5A9x0= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtg8u7uZrI0hoPbbZj5mH/sRNnx6SWjde0GamNdoFafJf5+bKIYtjqXT3MG6SDQuE/1xZlt4pWKIyEHtov3+RI= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T13:59:30.998 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T13:59:30.999 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T13:59:30.999 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T13:59:31.001 INFO:teuthology.task.internal:Checking packages... 2024-03-25T13:59:31.028 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T13:59:31.029 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T13:59:31.029 INFO:teuthology.packaging:ref: None 2024-03-25T13:59:31.029 INFO:teuthology.packaging:tag: None 2024-03-25T13:59:31.029 INFO:teuthology.packaging:branch: main 2024-03-25T13:59:31.029 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T13:59:31.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T13:59:31.252 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T13:59:31.254 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T13:59:31.260 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T13:59:31.260 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T13:59:31.287 INFO:teuthology.task.internal:Saving configuration 2024-03-25T13:59:31.304 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T13:59:31.311 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T13:59:31.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:54:30.875764', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8uAacY53DXOKDITja8/1iob9sZhkTWetwQMGMhL6oONG+hzEmsexaheGDGSltZaL6A1lH72N2TmBFw8y5A9x0='} 2024-03-25T13:59:31.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:54:30.877208', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtg8u7uZrI0hoPbbZj5mH/sRNnx6SWjde0GamNdoFafJf5+bKIYtjqXT3MG6SDQuE/1xZlt4pWKIyEHtov3+RI='} 2024-03-25T13:59:31.361 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T13:59:31.368 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T13:59:31.368 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T13:59:31.368 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T13:59:31.482 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f16892810d0>, signals=[15]) 2024-03-25T13:59:31.483 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T13:59:31.537 INFO:teuthology.task.internal:Opening connections... 2024-03-25T13:59:31.538 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T13:59:31.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T13:59:31.655 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T13:59:31.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T13:59:31.746 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T13:59:31.753 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-03-25T13:59:31.770 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-03-25T13:59:31.770 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T13:59:31.830 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-03-25T13:59:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-03-25T13:59:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T13:59:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-03-25T13:59:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T13:59:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T13:59:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T13:59:31.832 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-03-25T13:59:31.852 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-25T13:59:31.870 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-25T13:59:31.870 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-25T13:59:31.926 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T13:59:31.927 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T13:59:31.928 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-25T13:59:31.944 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T13:59:31.953 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T13:59:31.979 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T13:59:31.979 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-03-25T13:59:31.982 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-25T13:59:31.998 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T13:59:32.020 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T13:59:32.021 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-03-25T13:59:32.043 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-25T13:59:32.134 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T13:59:32.189 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T13:59:32.204 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T13:59:32.204 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T13:59:32.205 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T13:59:32.205 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T13:59:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T13:59:32.222 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-25T13:59:32.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T13:59:32.224 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-25T13:59:32.279 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-03-25T13:59:32.280 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-03-25T13:59:32.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-03-25T13:59:32.287 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-361.el9.x86_64 2024-03-25T13:59:32.287 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.14.0-361.el9.x86_64 2024-03-25T13:59:32.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-03-25T13:59:34.663 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-25T13:59:34.704 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.7 MB 00:00 2024-03-25T13:59:36.109 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-25T13:59:36.121 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T13:59:37.861 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-03-25T13:59:37.870 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-25T13:59:39.730 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-25T13:59:39.820 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-25T13:59:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-03-25T13:59:46.296 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-25T13:59:47.846 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-25T13:59:47.873 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-25T13:59:49.384 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T13:59:49.960 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T13:59:54.339 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-03-25T13:59:55.108 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T13:59:56.463 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T13:59:56.463 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T13:59:56.545 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T13:59:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T13:59:56.556 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-25T13:59:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T13:59:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T13:59:56.556 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-03-25T13:59:56.557 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T13:59:56.558 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 76 M 2024-03-25T13:59:56.558 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-03-25T13:59:56.558 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T13:59:56.732 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 389 kB/s | 44 kB 00:00 2024-03-25T13:59:57.365 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T13:59:57.365 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T13:59:57.470 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T13:59:57.479 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T13:59:57.479 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-03-25T13:59:57.480 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T13:59:57.481 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 76 M 2024-03-25T13:59:57.481 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 123 M 2024-03-25T13:59:57.481 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T13:59:57.726 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 513 kB/s | 44 kB 00:00 2024-03-25T13:59:58.109 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 32 MB/s | 15 MB 00:00 2024-03-25T13:59:58.501 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 35 MB/s | 27 MB 00:00 2024-03-25T13:59:58.818 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-25T13:59:58.819 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T13:59:58.819 INFO:teuthology.orchestra.run.smithi063.stdout:Total 57 MB/s | 76 MB 00:01 2024-03-25T13:59:59.510 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T13:59:59.937 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T13:59:59.937 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:00:00.616 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 3.7 MB/s | 15 MB 00:03 2024-03-25T14:00:00.908 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.8 MB/s | 33 MB 00:04 2024-03-25T14:00:00.991 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:00:00.991 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:00:03.608 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:00:04.273 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:00:04.359 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:00:05.999 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 3.0 MB/s | 27 MB 00:09 2024-03-25T14:00:05.999 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:00:05.999 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.0 MB/s | 76 MB 00:09 2024-03-25T14:00:06.665 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:00:07.061 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:00:07.062 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:00:07.211 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:00:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:00:08.065 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:00:08.066 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:00:10.532 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:00:11.336 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:00:11.432 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:00:12.454 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:00:12.512 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:14.188 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:00:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:00:17.640 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:19.071 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:00:19.235 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:24.003 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:48.571 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:48.571 INFO:teuthology.orchestra.run.smithi063.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T14:00:48.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:00:48.623 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:49.006 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:49.006 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:00:49.006 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:00:49.006 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T14:00:49.440 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:00:49.441 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:00:49.774 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:00:49.878 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:00:49.879 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:00:49.880 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T14:00:49.880 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T14:00:49.880 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T14:00:49.880 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T14:00:49.880 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T14:00:49.880 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:00:49.963 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:00:49.964 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:00:49.964 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.06 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 68.el9 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Group : Unspecified 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 8264254 2024-03-25T14:00:50.007 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : builder@centos.org 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:00:50.008 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T14:00:50.009 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T14:00:50.009 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T14:00:50.009 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-03-25T14:00:50.009 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:00:50.009 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T14:00:50.010 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T14:00:50.010 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 5.14.0-430.el9.x86_64 2024-03-25T14:00:50.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T14:00:50.617 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-03-25T14:00:51.938 INFO:teuthology.orchestra.run.smithi063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T14:00:51.986 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-03-25T14:00:51.989 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-25T14:00:52.008 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.kFPPBPDdfa 2024-03-25T14:00:52.118 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kFPPBPDdfa 2024-03-25T14:00:52.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.kFPPBPDdfa 2024-03-25T14:00:52.295 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.kFPPBPDdfa is 6KB 2024-03-25T14:00:52.345 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.kFPPBPDdfa 2024-03-25T14:00:52.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T14:00:52.428 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T14:00:52.429 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T14:00:52.429 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T14:00:52.429 INFO:teuthology.orchestra.run.smithi063.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T14:00:52.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T14:00:52.576 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-03-25T14:00:53.401 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:53.401 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T14:00:53.402 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:00:53.446 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T14:00:54.242 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T14:00:54.242 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:00:54.242 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:00:54.242 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T14:00:54.242 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T14:00:54.242 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T14:00:54.242 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T14:00:54.243 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:00:54.243 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:00:54.545 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:00:54.643 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:00:54.643 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:00:54.643 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T14:00:54.643 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T14:00:54.644 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T14:00:54.644 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T14:00:54.644 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T14:00:54.644 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T14:00:54.723 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T14:00:54.723 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T14:00:54.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-03-25T14:00:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T14:00:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T14:00:54.763 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T14:00:54.763 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-03-25T14:00:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:00:54.763 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T14:00:54.764 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T14:00:54.764 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-430.el9.x86_64 2024-03-25T14:00:54.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T14:00:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-25T14:00:56.869 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T14:00:56.905 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-25T14:00:56.907 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-25T14:00:56.923 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.KEvl6usAmj 2024-03-25T14:00:56.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KEvl6usAmj 2024-03-25T14:00:56.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.KEvl6usAmj 2024-03-25T14:00:57.133 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.KEvl6usAmj is 6KB 2024-03-25T14:00:57.182 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.KEvl6usAmj 2024-03-25T14:00:57.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T14:00:57.262 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T14:00:57.262 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T14:00:57.263 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T14:00:57.263 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T14:00:57.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T14:00:57.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-25T14:01:22.603 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T14:01:22.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T14:01:22.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:01:27.396 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T14:01:27.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T14:01:27.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:01:57.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-25T14:02:01.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-25T14:02:06.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T14:02:06.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:02:10.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T14:02:10.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:02:24.671 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-03-25T14:02:29.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-25T14:02:36.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T14:02:36.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:02:37.147 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-25T14:02:37.583 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T14:02:37.583 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T14:02:37.583 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-03-25T14:02:37.597 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-430.el9.x86_64 2024-03-25T14:02:37.598 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T14:02:37.598 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T14:02:37.598 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T14:02:38.598 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T14:02:38.599 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T14:02:38.671 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-03-25T14:02:38.738 DEBUG:teuthology.parallel:result is None 2024-03-25T14:02:41.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T14:02:41.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:02:44.335 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-25T14:02:44.833 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T14:02:44.833 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T14:02:44.833 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-25T14:02:44.847 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-430.el9.x86_64 2024-03-25T14:02:44.847 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T14:02:44.848 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T14:02:44.848 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T14:02:45.848 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T14:02:45.848 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T14:02:45.951 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-25T14:02:46.001 DEBUG:teuthology.parallel:result is None 2024-03-25T14:02:46.002 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T14:02:46.008 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T14:02:46.009 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T14:02:46.012 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T14:02:46.027 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T14:02:46.042 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T14:02:46.076 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T14:02:46.076 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T14:02:46.079 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T14:02:46.106 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T14:02:46.112 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T14:02:46.112 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T14:02:46.143 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T14:02:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:02:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:02:46.197 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:02:46.210 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T14:02:46.212 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T14:02:46.219 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T14:02:46.219 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T14:02:46.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T14:02:46.259 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T14:02:46.268 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T14:02:46.268 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T14:02:46.302 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T14:02:46.317 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:02:46.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:02:46.458 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:02:46.497 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:02:46.542 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:02:46.542 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T14:02:46.612 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:02:46.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T14:02:46.701 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:02:46.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T14:02:46.824 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:02:46.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T14:02:46.896 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-03-25T14:02:46.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-25T14:02:46.953 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:02:46.985 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:02:47.402 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T14:02:47.409 INFO:teuthology.task.internal:Starting timer... 2024-03-25T14:02:47.409 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T14:02:47.446 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T14:02:47.478 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-03-25T14:02:47.553 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-03-25T14:02:47.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-25T14:02:47.606 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-25T14:02:47.610 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T14:02:47.611 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-03-25T14:02:47.634 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-03-25T14:02:47.635 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-25T14:02:47.654 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-25T14:02:47.655 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-03-25T14:02:47.655 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:02:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:02:47.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:02:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:02:47.730 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T14:02:47.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-03-25T14:02:47.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-25T14:02:47.815 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T14:02:47.825 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T14:02:47.830 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T14:02:47.963 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T14:02:47.979 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T14:02:47.981 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T14:07:08.846 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-03-25T14:07:08.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T14:07:08.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:07:08.941 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-25T14:07:09.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-25T14:07:09.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T14:07:09.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T14:07:09.107 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-25T14:07:09.185 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T14:07:09.185 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T14:07:09.198 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T14:07:09.198 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T14:07:09.199 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:07:09.202 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T14:07:09.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:07:09.229 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T14:07:09.246 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T14:07:09.257 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T14:07:09.273 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T14:07:09.275 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-03-25T14:07:09.288 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-03-25T14:07:09.302 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-25T14:07:09.304 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T14:07:09.316 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-25T14:07:09.321 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T14:07:09.332 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T14:07:09.349 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T14:07:09.374 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:07:09.379 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:07:09.379 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-25T14:07:09.379 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.379 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.379 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.380 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.402 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:07:09.405 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:07:09.405 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-25T14:07:09.405 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.405 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.405 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.405 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:07:09.406 INFO:teuthology.run_tasks:Running task install... 2024-03-25T14:07:09.414 DEBUG:teuthology.task.install:project ceph 2024-03-25T14:07:09.414 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T14:07:09.414 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T14:07:09.414 INFO:teuthology.task.install:Using flavor: default 2024-03-25T14:07:09.422 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T14:07:09.422 INFO:teuthology.task.install:extra packages: [] 2024-03-25T14:07:09.423 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T14:07:09.423 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:07:09.425 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T14:07:09.426 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:07:09.734 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T14:07:09.734 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T14:07:09.748 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T14:07:09.748 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T14:07:09.851 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T14:07:09.851 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:07:09.851 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T14:07:09.860 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T14:07:09.860 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:07:09.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T14:07:09.879 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T14:07:09.879 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T14:07:09.889 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T14:07:09.889 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T14:07:09.947 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T14:07:09.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T14:07:10.027 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T14:07:10.037 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T14:07:10.093 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-03-25T14:07:10.094 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-03-25T14:07:10.103 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-25T14:07:10.105 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-25T14:07:10.347 INFO:teuthology.orchestra.run.smithi063.stdout:74 files removed 2024-03-25T14:07:10.350 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-25T14:07:10.374 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-03-25T14:07:10.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-03-25T14:07:11.004 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 252 kB/s | 82 kB 00:00 2024-03-25T14:07:11.086 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 202 kB/s | 82 kB 00:00 2024-03-25T14:07:11.453 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-25T14:07:11.454 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-25T14:07:11.812 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-25T14:07:11.814 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-25T14:07:12.373 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T14:07:12.474 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T14:07:13.815 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-25T14:07:13.933 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-25T14:07:14.589 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-25T14:07:15.168 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-03-25T14:07:20.850 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 19 MB 00:04 2024-03-25T14:07:31.614 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-25T14:07:32.500 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 763 kB/s | 6.1 MB 00:08 2024-03-25T14:07:34.134 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T14:07:36.405 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-25T14:07:41.327 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T14:07:43.897 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:07:43.915 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:07:43.915 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:07:43.915 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T14:07:43.916 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T14:07:43.917 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T14:07:43.918 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout:Install 30 Packages 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-03-25T14:07:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:07:43.920 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-03-25T14:07:43.920 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:07:44.056 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 716 kB/s | 6.1 MB 00:08 2024-03-25T14:07:44.970 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 6.9 MB/s | 5.3 MB 00:00 2024-03-25T14:07:45.070 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T14:07:45.571 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.1 MB/s | 9.7 MB 00:01 2024-03-25T14:07:45.604 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.4 MB/s | 742 kB 00:00 2024-03-25T14:07:45.654 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.8 MB/s | 498 kB 00:00 2024-03-25T14:07:45.721 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 678 kB/s | 45 kB 00:00 2024-03-25T14:07:45.740 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-25T14:07:45.805 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-25T14:07:46.109 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 10 MB/s | 20 MB 00:01 2024-03-25T14:07:46.139 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 485 kB/s | 162 kB 00:00 2024-03-25T14:07:46.164 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 5.7 MB/s | 320 kB 00:00 2024-03-25T14:07:46.248 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 7.9 MB/s | 5.1 MB 00:00 2024-03-25T14:07:46.273 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.2 MB/s | 298 kB 00:00 2024-03-25T14:07:46.299 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 742 kB/s | 99 kB 00:00 2024-03-25T14:07:46.432 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-03-25T14:07:46.457 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 866 kB/s | 159 kB 00:00 2024-03-25T14:07:46.483 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 328 kB/s | 44 kB 00:00 2024-03-25T14:07:46.516 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-25T14:07:46.541 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 719 kB/s | 41 kB 00:00 2024-03-25T14:07:46.567 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-25T14:07:46.742 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 174 kB/s | 35 kB 00:00 2024-03-25T14:07:46.776 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 213 kB/s | 44 kB 00:00 2024-03-25T14:07:46.801 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 761 kB/s | 216 kB 00:00 2024-03-25T14:07:46.817 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T14:07:47.043 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 107 kB/s | 26 kB 00:00 2024-03-25T14:07:47.235 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 627 kB/s | 309 kB 00:00 2024-03-25T14:07:47.335 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-25T14:07:47.360 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 153 kB/s | 49 kB 00:00 2024-03-25T14:07:47.610 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-03-25T14:07:47.636 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 696 kB/s | 192 kB 00:00 2024-03-25T14:07:47.670 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 838 kB 00:00 2024-03-25T14:07:48.078 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.7 MB/s | 3.1 MB 00:00 2024-03-25T14:07:48.362 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-25T14:07:49.004 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.5 MB/s | 3.4 MB 00:01 2024-03-25T14:07:49.008 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:07:49.009 INFO:teuthology.orchestra.run.smithi153.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-25T14:07:49.038 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:07:49.074 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:07:49.074 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:07:49.552 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:07:49.553 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:07:50.991 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:07:51.067 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T14:07:51.100 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T14:07:51.126 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T14:07:51.148 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:07:51.164 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:07:51.367 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:07:51.420 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:07:51.741 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T14:07:51.819 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:07:51.911 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:07:52.001 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T14:07:52.057 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T14:07:52.087 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:07:52.150 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:07:52.179 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T14:07:52.231 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T14:07:52.300 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:07:52.323 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:07:52.369 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:07:52.396 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T14:07:52.515 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T14:07:52.556 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T14:07:52.612 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T14:07:52.646 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T14:07:52.674 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T14:07:52.707 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T14:07:53.030 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T14:07:53.062 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:07:53.152 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:07:53.184 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T14:07:53.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T14:07:53.680 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T14:07:53.715 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:07:53.786 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:07:53.862 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T14:07:53.892 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T14:07:54.523 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:07:54.530 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:07:54.530 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:07:54.530 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:07:54.530 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T14:07:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T14:07:54.532 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout:Install 30 Packages 2024-03-25T14:07:54.533 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-03-25T14:07:54.534 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:07:54.534 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 58 M 2024-03-25T14:07:54.534 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:07:56.106 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:07:56.164 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:07:56.252 INFO:teuthology.orchestra.run.smithi063.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.7 MB/s | 5.3 MB 00:01 2024-03-25T14:07:56.461 INFO:teuthology.orchestra.run.smithi063.stdout:(2/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 12 MB/s | 20 MB 00:01 2024-03-25T14:07:56.486 INFO:teuthology.orchestra.run.smithi063.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 108 kB/s | 25 kB 00:00 2024-03-25T14:07:56.544 INFO:teuthology.orchestra.run.smithi063.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-25T14:07:56.619 INFO:teuthology.orchestra.run.smithi063.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.6 MB/s | 498 kB 00:00 2024-03-25T14:07:56.709 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:07:56.711 INFO:teuthology.orchestra.run.smithi063.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 492 kB/s | 45 kB 00:00 2024-03-25T14:07:56.750 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:07:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:07:56.842 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T14:07:56.842 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T14:07:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:07:56.854 INFO:teuthology.orchestra.run.smithi063.stdout:(7/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.8 MB/s | 9.7 MB 00:02 2024-03-25T14:07:56.876 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:07:56.879 INFO:teuthology.orchestra.run.smithi063.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 786 kB/s | 131 kB 00:00 2024-03-25T14:07:56.954 INFO:teuthology.orchestra.run.smithi063.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 12 MB/s | 5.1 MB 00:00 2024-03-25T14:07:56.979 INFO:teuthology.orchestra.run.smithi063.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.3 MB/s | 162 kB 00:00 2024-03-25T14:07:57.005 INFO:teuthology.orchestra.run.smithi063.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.5 MB/s | 320 kB 00:00 2024-03-25T14:07:57.030 INFO:teuthology.orchestra.run.smithi063.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-25T14:07:57.055 INFO:teuthology.orchestra.run.smithi063.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-25T14:07:57.181 INFO:teuthology.orchestra.run.smithi063.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2024-03-25T14:07:57.206 INFO:teuthology.orchestra.run.smithi063.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 44 kB 00:00 2024-03-25T14:07:57.231 INFO:teuthology.orchestra.run.smithi063.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 794 kB/s | 159 kB 00:00 2024-03-25T14:07:57.340 INFO:teuthology.orchestra.run.smithi063.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 661 kB 00:00 2024-03-25T14:07:57.365 INFO:teuthology.orchestra.run.smithi063.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-25T14:07:57.407 INFO:teuthology.orchestra.run.smithi063.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 235 kB/s | 41 kB 00:00 2024-03-25T14:07:58.007 INFO:teuthology.orchestra.run.smithi063.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 58 kB/s | 35 kB 00:00 2024-03-25T14:07:58.099 INFO:teuthology.orchestra.run.smithi063.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 64 kB/s | 44 kB 00:00 2024-03-25T14:07:58.308 INFO:teuthology.orchestra.run.smithi063.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 224 kB/s | 216 kB 00:00 2024-03-25T14:07:58.333 INFO:teuthology.orchestra.run.smithi063.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 949 kB/s | 309 kB 00:00 2024-03-25T14:07:58.583 INFO:teuthology.orchestra.run.smithi063.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 94 kB/s | 26 kB 00:00 2024-03-25T14:07:58.609 INFO:teuthology.orchestra.run.smithi063.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 176 kB/s | 49 kB 00:00 2024-03-25T14:07:58.709 INFO:teuthology.orchestra.run.smithi063.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 539 kB/s | 67 kB 00:00 2024-03-25T14:07:59.018 INFO:teuthology.orchestra.run.smithi063.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 622 kB/s | 192 kB 00:00 2024-03-25T14:07:59.359 INFO:teuthology.orchestra.run.smithi063.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 838 kB 00:00 2024-03-25T14:07:59.493 INFO:teuthology.orchestra.run.smithi063.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-03-25T14:07:59.593 INFO:teuthology.orchestra.run.smithi063.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2024-03-25T14:07:59.794 INFO:teuthology.orchestra.run.smithi063.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-03-25T14:08:00.069 INFO:teuthology.orchestra.run.smithi063.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.8 MB/s | 3.4 MB 00:00 2024-03-25T14:08:00.076 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:00.076 INFO:teuthology.orchestra.run.smithi063.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-25T14:08:00.109 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:08:00.151 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:08:00.151 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:08:00.660 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:08:00.661 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:08:02.284 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:08:02.373 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T14:08:02.406 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T14:08:02.432 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T14:08:02.463 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:08:02.480 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T14:08:02.659 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:08:02.711 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T14:08:03.063 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T14:08:03.117 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:08:03.193 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:08:03.318 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T14:08:03.344 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:08:03.406 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T14:08:03.436 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T14:08:03.480 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T14:08:03.550 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:08:03.571 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:08:03.617 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T14:08:03.645 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T14:08:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T14:08:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T14:08:03.820 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T14:08:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T14:08:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T14:08:03.914 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T14:08:04.243 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T14:08:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:08:04.360 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T14:08:04.391 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T14:08:04.432 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T14:08:04.908 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T14:08:04.931 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:08:05.002 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T14:08:05.071 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T14:08:05.099 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-25T14:08:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:07.329 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:08:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:08:07.485 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:08:07.510 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:08:07.510 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:07.510 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T14:08:07.510 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:08:07.510 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:08:07.510 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:07.512 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:08:07.598 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:08:07.598 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:08:07.936 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T14:08:07.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:08:08.065 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T14:08:08.065 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T14:08:08.065 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T14:08:08.065 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:08.100 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:08:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:08:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T14:08:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T14:08:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T14:08:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T14:08:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T14:08:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:08:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T14:08:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T14:08:09.790 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:08:09.790 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:09.790 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-25T14:08:09.790 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.790 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.790 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:08:09.790 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.791 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T14:08:09.792 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T14:08:09.793 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T14:08:09.793 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T14:08:09.793 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T14:08:09.793 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:09.793 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:10.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-03-25T14:08:10.745 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:29 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:10.894 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:10.894 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T14:08:10.895 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:10.896 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:08:10.896 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:10.896 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-25T14:08:10.896 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:10.896 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 48 M 2024-03-25T14:08:10.897 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 204 M 2024-03-25T14:08:10.897 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:08:11.137 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-25T14:08:11.188 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T14:08:11.221 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T14:08:11.304 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 449 kB/s | 163 kB 00:00 2024-03-25T14:08:12.054 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 224 kB/s | 187 kB 00:00 2024-03-25T14:08:13.430 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-25T14:08:13.434 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:13.434 INFO:teuthology.orchestra.run.smithi153.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-25T14:08:13.448 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:08:13.468 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:08:13.855 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:08:13.856 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:08:14.439 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:08:14.514 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T14:08:14.552 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T14:08:14.592 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:08:14.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T14:08:14.658 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:08:18.319 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-03-25T14:08:18.320 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:19.128 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:08:19.199 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:08:19.260 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:08:19.284 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T14:08:19.285 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:19.285 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T14:08:19.285 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:08:19.285 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T14:08:19.285 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:19.286 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:08:19.371 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T14:08:19.371 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:08:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:08:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T14:08:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T14:08:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T14:08:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T14:08:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T14:08:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T14:08:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T14:08:21.604 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T14:08:21.604 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:21.604 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-03-25T14:08:21.604 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.604 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.604 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:08:21.604 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.605 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T14:08:21.606 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T14:08:21.607 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T14:08:21.607 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T14:08:21.607 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:21.607 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:08:21.714 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:08:21.714 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T14:08:21.714 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T14:08:21.714 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:08:21.714 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T14:08:21.714 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T14:08:21.960 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T14:08:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T14:08:22.093 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:22.093 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:22.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-03-25T14:08:22.529 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:31 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:08:22.689 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:08:22.690 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:22.690 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:08:22.690 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:22.690 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:08:22.690 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T14:08:22.690 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-03-25T14:08:22.691 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:22.692 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 48 M 2024-03-25T14:08:22.692 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 204 M 2024-03-25T14:08:22.692 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:08:23.018 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:42 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:23.180 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:23.182 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-25T14:08:23.182 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:08:23.182 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-25T14:08:23.182 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:08:23.182 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T14:08:23.182 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T14:08:23.183 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T14:08:23.184 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout:Install 36 Packages 2024-03-25T14:08:23.185 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:23.186 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 30 M 2024-03-25T14:08:23.186 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 107 M 2024-03-25T14:08:23.186 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:08:23.187 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 631 kB/s | 163 kB 00:00 2024-03-25T14:08:23.279 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 618 kB/s | 217 kB 00:00 2024-03-25T14:08:23.329 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-03-25T14:08:23.354 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 835 kB/s | 63 kB 00:00 2024-03-25T14:08:23.904 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 325 kB/s | 187 kB 00:00 2024-03-25T14:08:24.198 INFO:teuthology.orchestra.run.smithi153.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 26 kB/s | 6.6 kB 00:00 2024-03-25T14:08:25.057 INFO:teuthology.orchestra.run.smithi153.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-03-25T14:08:25.329 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-25T14:08:25.332 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:25.333 INFO:teuthology.orchestra.run.smithi063.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-25T14:08:25.347 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:08:25.369 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:08:25.370 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:08:25.424 INFO:teuthology.orchestra.run.smithi153.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 3.8 MB/s | 4.7 MB 00:01 2024-03-25T14:08:25.474 INFO:teuthology.orchestra.run.smithi153.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.9 MB/s | 247 kB 00:00 2024-03-25T14:08:25.541 INFO:teuthology.orchestra.run.smithi153.stdout:(5/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 954 kB/s | 1.5 MB 00:01 2024-03-25T14:08:25.608 INFO:teuthology.orchestra.run.smithi153.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T14:08:25.633 INFO:teuthology.orchestra.run.smithi153.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-25T14:08:25.650 INFO:teuthology.orchestra.run.smithi153.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2024-03-25T14:08:25.667 INFO:teuthology.orchestra.run.smithi153.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T14:08:25.684 INFO:teuthology.orchestra.run.smithi153.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T14:08:25.710 INFO:teuthology.orchestra.run.smithi153.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T14:08:25.760 INFO:teuthology.orchestra.run.smithi153.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T14:08:25.766 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:08:25.766 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:08:25.785 INFO:teuthology.orchestra.run.smithi153.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-25T14:08:25.810 INFO:teuthology.orchestra.run.smithi153.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T14:08:25.836 INFO:teuthology.orchestra.run.smithi153.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T14:08:25.928 INFO:teuthology.orchestra.run.smithi153.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-25T14:08:26.103 INFO:teuthology.orchestra.run.smithi153.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 428 kB/s | 125 kB 00:00 2024-03-25T14:08:26.170 INFO:teuthology.orchestra.run.smithi153.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 625 kB/s | 151 kB 00:00 2024-03-25T14:08:26.195 INFO:teuthology.orchestra.run.smithi153.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 473 kB/s | 43 kB 00:00 2024-03-25T14:08:26.357 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:08:26.362 INFO:teuthology.orchestra.run.smithi153.stdout:(20/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 17 MB 00:01 2024-03-25T14:08:26.447 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T14:08:26.488 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T14:08:26.528 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:08:26.572 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T14:08:26.599 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:08:26.662 INFO:teuthology.orchestra.run.smithi153.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 349 kB/s | 172 kB 00:00 2024-03-25T14:08:27.237 INFO:teuthology.orchestra.run.smithi153.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 12 kB/s | 11 kB 00:00 2024-03-25T14:08:27.638 INFO:teuthology.orchestra.run.smithi153.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 248 kB/s | 358 kB 00:01 2024-03-25T14:08:27.663 INFO:teuthology.orchestra.run.smithi153.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 18 kB/s | 18 kB 00:01 2024-03-25T14:08:27.730 INFO:teuthology.orchestra.run.smithi153.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 47 kB/s | 23 kB 00:00 2024-03-25T14:08:27.772 INFO:teuthology.orchestra.run.smithi153.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-03-25T14:08:27.814 INFO:teuthology.orchestra.run.smithi153.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 131 kB/s | 20 kB 00:00 2024-03-25T14:08:27.889 INFO:teuthology.orchestra.run.smithi153.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 292 kB/s | 46 kB 00:00 2024-03-25T14:08:28.106 INFO:teuthology.orchestra.run.smithi153.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 236 kB/s | 79 kB 00:00 2024-03-25T14:08:28.507 INFO:teuthology.orchestra.run.smithi153.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 393 kB/s | 272 kB 00:00 2024-03-25T14:08:28.531 INFO:teuthology.orchestra.run.smithi153.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 26 kB/s | 16 kB 00:00 2024-03-25T14:08:28.599 INFO:teuthology.orchestra.run.smithi153.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 183 kB/s | 90 kB 00:00 2024-03-25T14:08:28.632 INFO:teuthology.orchestra.run.smithi153.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 284 kB/s | 36 kB 00:00 2024-03-25T14:08:28.832 INFO:teuthology.orchestra.run.smithi153.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 764 kB/s | 230 kB 00:00 2024-03-25T14:08:28.974 INFO:teuthology.orchestra.run.smithi153.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 58 kB/s | 20 kB 00:00 2024-03-25T14:08:29.149 INFO:teuthology.orchestra.run.smithi153.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 776 kB/s | 427 kB 00:00 2024-03-25T14:08:29.154 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:29.154 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.0 MB/s | 30 MB 00:05 2024-03-25T14:08:29.177 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:08:29.221 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:08:29.221 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:08:29.636 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:08:29.637 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:08:30.951 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:08:31.010 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T14:08:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T14:08:31.105 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T14:08:31.174 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T14:08:31.211 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T14:08:31.244 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T14:08:31.311 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T14:08:31.336 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T14:08:31.384 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:08:31.430 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T14:08:31.471 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T14:08:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T14:08:31.598 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T14:08:31.636 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T14:08:31.685 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T14:08:31.740 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T14:08:31.926 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T14:08:31.974 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T14:08:32.012 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T14:08:32.046 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T14:08:32.068 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T14:08:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T14:08:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:08:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T14:08:32.261 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T14:08:32.295 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T14:08:32.330 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T14:08:32.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T14:08:32.452 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T14:08:32.524 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T14:08:32.670 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T14:08:32.803 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:08:33.002 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T14:08:33.033 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:08:33.058 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:08:33.059 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:33.059 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T14:08:33.059 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:08:33.059 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:08:33.059 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:33.792 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T14:08:33.792 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T14:08:33.792 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T14:08:33.792 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T14:08:33.793 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T14:08:33.793 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T14:08:34.176 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T14:08:34.176 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:34.176 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:08:34.176 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:34.176 INFO:teuthology.orchestra.run.smithi063.stdout: jq-1.6-16.el9.x86_64 2024-03-25T14:08:34.176 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:34.176 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T14:08:34.177 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T14:08:34.177 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T14:08:34.177 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:34.177 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:08:34.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-03-25T14:08:34.509 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:08:34.536 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:08:34.536 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:34.536 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T14:08:34.536 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:08:34.536 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:08:34.536 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:34.976 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:08:34.999 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:08:35.000 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:35.000 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T14:08:35.000 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:08:35.000 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:08:35.000 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:35.035 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:44 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:08:35.195 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:08:35.197 INFO:teuthology.orchestra.run.smithi063.stdout:========================================================================================= 2024-03-25T14:08:35.197 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:08:35.197 INFO:teuthology.orchestra.run.smithi063.stdout:========================================================================================= 2024-03-25T14:08:35.197 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T14:08:35.198 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T14:08:35.199 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T14:08:35.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T14:08:35.201 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:35.201 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:08:35.201 INFO:teuthology.orchestra.run.smithi063.stdout:========================================================================================= 2024-03-25T14:08:35.201 INFO:teuthology.orchestra.run.smithi063.stdout:Install 36 Packages 2024-03-25T14:08:35.201 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:35.201 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 30 M 2024-03-25T14:08:35.202 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 107 M 2024-03-25T14:08:35.202 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:08:35.202 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:08:35.224 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:08:35.224 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:35.225 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T14:08:35.225 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:08:35.225 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:08:35.225 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:35.283 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:08:36.007 INFO:teuthology.orchestra.run.smithi063.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-03-25T14:08:36.115 INFO:teuthology.orchestra.run.smithi063.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-03-25T14:08:36.565 INFO:teuthology.orchestra.run.smithi063.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 2.1 MB 00:00 2024-03-25T14:08:36.665 INFO:teuthology.orchestra.run.smithi063.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-25T14:08:36.782 INFO:teuthology.orchestra.run.smithi063.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T14:08:36.824 INFO:teuthology.orchestra.run.smithi063.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T14:08:36.858 INFO:teuthology.orchestra.run.smithi063.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-25T14:08:36.891 INFO:teuthology.orchestra.run.smithi063.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T14:08:36.925 INFO:teuthology.orchestra.run.smithi063.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T14:08:37.008 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T14:08:37.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:08:37.010 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T14:08:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T14:08:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T14:08:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T14:08:37.011 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T14:08:37.084 INFO:teuthology.orchestra.run.smithi063.stdout:(10/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 17 MB/s | 17 MB 00:00 2024-03-25T14:08:37.109 INFO:teuthology.orchestra.run.smithi063.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 241 kB 00:00 2024-03-25T14:08:37.176 INFO:teuthology.orchestra.run.smithi063.stdout:(12/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-03-25T14:08:37.209 INFO:teuthology.orchestra.run.smithi063.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 9.2 MB/s | 1.1 MB 00:00 2024-03-25T14:08:37.234 INFO:teuthology.orchestra.run.smithi063.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 825 kB/s | 103 kB 00:00 2024-03-25T14:08:37.260 INFO:teuthology.orchestra.run.smithi063.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-25T14:08:37.285 INFO:teuthology.orchestra.run.smithi063.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T14:08:37.402 INFO:teuthology.orchestra.run.smithi063.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-03-25T14:08:37.544 INFO:teuthology.orchestra.run.smithi063.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-03-25T14:08:37.669 INFO:teuthology.orchestra.run.smithi063.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 393 kB/s | 151 kB 00:00 2024-03-25T14:08:37.803 INFO:teuthology.orchestra.run.smithi063.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 108 kB/s | 43 kB 00:00 2024-03-25T14:08:37.887 INFO:teuthology.orchestra.run.smithi063.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 127 kB/s | 11 kB 00:00 2024-03-25T14:08:37.912 INFO:teuthology.orchestra.run.smithi063.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-25T14:08:37.962 INFO:teuthology.orchestra.run.smithi063.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 237 kB/s | 18 kB 00:00 2024-03-25T14:08:37.979 INFO:teuthology.orchestra.run.smithi063.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T14:08:38.004 INFO:teuthology.orchestra.run.smithi063.stdout:(25/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 373 kB/s | 172 kB 00:00 2024-03-25T14:08:38.038 INFO:teuthology.orchestra.run.smithi063.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-25T14:08:38.055 INFO:teuthology.orchestra.run.smithi063.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-25T14:08:38.088 INFO:teuthology.orchestra.run.smithi063.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 551 kB/s | 46 kB 00:00 2024-03-25T14:08:38.135 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T14:08:38.135 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:38.135 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:08:38.135 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:38.135 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:38.135 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T14:08:38.136 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T14:08:38.137 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:38.138 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:38.289 INFO:teuthology.orchestra.run.smithi063.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 82 kB/s | 16 kB 00:00 2024-03-25T14:08:38.306 INFO:teuthology.orchestra.run.smithi063.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 295 kB/s | 79 kB 00:00 2024-03-25T14:08:38.331 INFO:teuthology.orchestra.run.smithi063.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 986 kB/s | 272 kB 00:00 2024-03-25T14:08:38.390 INFO:teuthology.orchestra.run.smithi063.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 894 kB/s | 90 kB 00:00 2024-03-25T14:08:38.406 INFO:teuthology.orchestra.run.smithi063.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 355 kB/s | 36 kB 00:00 2024-03-25T14:08:38.432 INFO:teuthology.orchestra.run.smithi063.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-25T14:08:38.474 INFO:teuthology.orchestra.run.smithi063.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-25T14:08:38.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-03-25T14:08:38.574 INFO:teuthology.orchestra.run.smithi063.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-03-25T14:08:38.579 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:38.579 INFO:teuthology.orchestra.run.smithi063.stdout:Total 8.8 MB/s | 30 MB 00:03 2024-03-25T14:08:38.602 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:08:38.646 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:08:38.646 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:08:39.061 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:08:39.061 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:08:39.069 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:58 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:39.165 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:08:39.217 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:39.219 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:08:39.219 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:39.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-03-25T14:08:39.849 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:58 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:39.998 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:39.999 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:39.999 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:08:39.999 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:39.999 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:08:39.999 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T14:08:39.999 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:40.000 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:08:40.000 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:40.000 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:08:40.000 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:40.000 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 747 k 2024-03-25T14:08:40.000 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 754 k 2024-03-25T14:08:40.000 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:08:40.290 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-25T14:08:40.291 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:40.291 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-25T14:08:40.291 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:08:40.296 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:08:40.297 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:08:40.302 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:08:40.302 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:08:40.374 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:08:40.515 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:08:40.601 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T14:08:40.642 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T14:08:40.679 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T14:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T14:08:40.757 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:40.837 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:40.920 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T14:08:40.952 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T14:08:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T14:08:41.131 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T14:08:41.172 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T14:08:41.245 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T14:08:41.273 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:41.298 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T14:08:41.336 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T14:08:41.393 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T14:08:41.449 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T14:08:41.565 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:41.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:41.565 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:08:41.565 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:08:41.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:41.566 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:41.640 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T14:08:41.690 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T14:08:41.730 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T14:08:41.763 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T14:08:41.793 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T14:08:41.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T14:08:41.872 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T14:08:42.028 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:08:42.070 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T14:08:42.110 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T14:08:42.146 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T14:08:42.179 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T14:08:42.273 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T14:08:42.310 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T14:08:42.377 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:01 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:42.386 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T14:08:42.525 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T14:08:42.529 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:42.530 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:08:42.530 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:42.530 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:08:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T14:08:42.530 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:42.531 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:08:42.531 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:08:42.531 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:08:42.531 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:42.531 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 145 k 2024-03-25T14:08:42.531 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 436 k 2024-03-25T14:08:42.531 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:08:42.658 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:08:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T14:08:42.865 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:08:42.888 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 407 kB/s | 145 kB 00:00 2024-03-25T14:08:42.889 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:42.889 INFO:teuthology.orchestra.run.smithi153.stdout:Total 406 kB/s | 145 kB 00:00 2024-03-25T14:08:42.889 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:08:42.889 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T14:08:42.889 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:42.890 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T14:08:42.890 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:08:42.890 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T14:08:42.890 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:42.896 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:08:42.897 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:08:42.948 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:08:42.948 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:08:43.088 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:08:43.156 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:08:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:08:43.180 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:43.180 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T14:08:43.180 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:44.173 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:08:44.174 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:44.174 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:08:44.174 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:44.174 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:44.174 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:44.399 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:08:44.424 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T14:08:44.424 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:44.424 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T14:08:44.425 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:08:44.425 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T14:08:44.425 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:44.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-03-25T14:08:44.883 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:08:44.906 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T14:08:44.906 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:44.906 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T14:08:44.906 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:08:44.906 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T14:08:44.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:45.055 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:04 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:45.117 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:08:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T14:08:45.139 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:45.139 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T14:08:45.139 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:08:45.139 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T14:08:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:45.152 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:08:45.206 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:45.207 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:08:45.207 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:08:45.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T14:08:45.836 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:04 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:46.001 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T14:08:46.002 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-25T14:08:46.003 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:46.004 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.9 M 2024-03-25T14:08:46.004 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-25T14:08:46.004 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:08:46.417 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 79 kB/s | 15 kB 00:00 2024-03-25T14:08:46.476 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 125 kB/s | 31 kB 00:00 2024-03-25T14:08:46.567 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.6 MB/s | 238 kB 00:00 2024-03-25T14:08:46.609 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.2 MB/s | 1.0 MB 00:00 2024-03-25T14:08:46.768 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.3 MB/s | 1.8 MB 00:00 2024-03-25T14:08:46.894 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-25T14:08:46.902 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T14:08:46.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T14:08:46.904 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T14:08:46.905 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T14:08:46.927 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T14:08:46.986 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-25T14:08:47.044 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 528 kB/s | 31 kB 00:00 2024-03-25T14:08:47.119 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-25T14:08:47.194 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.8 MB/s | 2.0 MB 00:00 2024-03-25T14:08:48.002 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T14:08:48.002 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:48.002 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:08:48.002 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:48.002 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:48.002 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T14:08:48.003 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T14:08:48.004 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:48.005 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:08:48.253 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 512 kB/s | 863 kB 00:01 2024-03-25T14:08:48.257 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:48.257 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.0 MB/s | 6.9 MB 00:02 2024-03-25T14:08:48.276 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:08:48.294 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:08:48.295 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:08:48.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-03-25T14:08:48.489 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:08:48.489 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:08:48.936 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:57 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:08:48.997 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:08:49.037 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:08:49.070 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T14:08:49.091 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:08:49.092 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:08:49.092 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:08:49.114 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T14:08:49.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-03-25T14:08:49.168 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T14:08:49.199 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T14:08:49.307 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T14:08:49.433 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T14:08:49.526 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T14:08:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T14:08:49.599 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:08:49.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T14:08:49.728 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:58 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:08:49.883 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T14:08:49.884 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:08:49.885 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:49.885 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:08:49.885 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:49.885 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:08:49.885 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T14:08:49.885 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:49.885 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:08:49.886 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:49.886 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:08:49.886 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:49.886 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 747 k 2024-03-25T14:08:49.886 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 754 k 2024-03-25T14:08:49.886 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:08:49.922 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:08:50.155 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.7 MB/s | 747 kB 00:00 2024-03-25T14:08:50.155 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:50.156 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.7 MB/s | 747 kB 00:00 2024-03-25T14:08:50.156 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:08:50.161 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:08:50.162 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:08:50.168 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:08:50.168 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:08:50.241 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:08:50.662 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:50.753 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:50.778 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:08:50.778 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T14:08:50.778 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T14:08:50.778 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T14:08:50.778 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T14:08:50.778 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T14:08:50.779 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T14:08:50.779 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T14:08:50.779 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T14:08:50.779 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:08:50.779 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T14:08:50.779 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T14:08:51.122 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T14:08:51.174 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T14:08:51.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T14:08:51.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T14:08:51.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T14:08:51.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T14:08:51.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T14:08:51.175 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:51.175 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:08:51.415 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T14:08:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:51.416 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:08:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:08:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:51.416 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:08:51.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T14:08:51.616 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T14:08:51.993 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:10 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:08:52.144 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:08:52.145 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-25T14:08:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:08:52.145 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-25T14:08:52.145 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:08:52.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T14:08:52.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T14:08:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:52.147 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:08:52.147 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-25T14:08:52.147 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-25T14:08:52.147 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:08:52.147 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-03-25T14:08:52.147 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-03-25T14:08:52.148 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:08:52.194 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:01 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:08:52.349 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:08:52.350 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:52.350 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:08:52.350 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:52.350 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:08:52.350 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T14:08:52.350 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:52.351 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:08:52.351 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:08:52.351 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:08:52.351 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:52.351 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 145 k 2024-03-25T14:08:52.351 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 436 k 2024-03-25T14:08:52.351 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:08:52.585 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 621 kB/s | 145 kB 00:00 2024-03-25T14:08:52.586 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:52.586 INFO:teuthology.orchestra.run.smithi063.stdout:Total 617 kB/s | 145 kB 00:00 2024-03-25T14:08:52.586 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:08:52.594 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:08:52.594 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:08:52.649 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:08:52.649 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:08:52.676 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 32 kB 00:00 2024-03-25T14:08:52.709 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-03-25T14:08:52.743 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T14:08:52.792 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:08:52.861 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:08:52.886 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:08:52.886 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:08:52.886 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T14:08:52.886 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:52.935 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T14:08:53.044 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.8 MB/s | 3.0 MB 00:00 2024-03-25T14:08:53.111 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-25T14:08:53.136 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T14:08:53.236 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 10 MB/s | 7.4 MB 00:00 2024-03-25T14:08:53.545 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-25T14:08:53.637 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-25T14:08:53.662 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 461 kB/s | 195 kB 00:00 2024-03-25T14:08:53.745 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-25T14:08:53.749 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:53.749 INFO:teuthology.orchestra.run.smithi153.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-25T14:08:53.790 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:08:53.814 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:08:53.814 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:08:53.875 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T14:08:53.875 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:53.875 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:08:53.875 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:08:53.875 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:53.875 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:08:54.114 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-03-25T14:08:54.201 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:08:54.201 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:08:54.686 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:03 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:08:54.786 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:08:54.842 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:08:54.843 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:08:54.843 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:08:54.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T14:08:54.926 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:08:55.010 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T14:08:55.041 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T14:08:55.137 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:08:55.169 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T14:08:55.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:08:55.377 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:08:55.462 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T14:08:55.479 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:04 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:08:55.634 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:08:55.635 INFO:teuthology.orchestra.run.smithi063.stdout:====================================================================================== 2024-03-25T14:08:55.635 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:08:55.635 INFO:teuthology.orchestra.run.smithi063.stdout:====================================================================================== 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T14:08:55.636 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T14:08:55.637 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T14:08:55.637 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T14:08:55.637 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:55.637 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:08:55.637 INFO:teuthology.orchestra.run.smithi063.stdout:====================================================================================== 2024-03-25T14:08:55.637 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-03-25T14:08:55.637 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:08:55.638 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 6.9 M 2024-03-25T14:08:55.638 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-03-25T14:08:55.638 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:08:56.144 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T14:08:56.247 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T14:08:56.514 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 58 kB/s | 15 kB 00:00 2024-03-25T14:08:56.539 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 108 kB/s | 31 kB 00:00 2024-03-25T14:08:56.681 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T14:08:56.714 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T14:08:56.764 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.5 MB/s | 1.8 MB 00:00 2024-03-25T14:08:56.973 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-03-25T14:08:57.073 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-03-25T14:08:57.308 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 925 kB/s | 548 kB 00:00 2024-03-25T14:08:57.332 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 558 kB/s | 144 kB 00:00 2024-03-25T14:08:57.408 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 306 kB/s | 31 kB 00:00 2024-03-25T14:08:57.474 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-25T14:08:57.490 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T14:08:57.641 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-03-25T14:08:57.647 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:08:57.647 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.4 MB/s | 6.9 MB 00:02 2024-03-25T14:08:57.665 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:08:57.684 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:08:57.684 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:08:57.864 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:08:57.864 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:08:58.386 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:08:58.467 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T14:08:58.502 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T14:08:58.519 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T14:08:58.550 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T14:08:58.550 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:08:58.579 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T14:08:58.690 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T14:08:58.825 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T14:08:58.918 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T14:08:58.976 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T14:08:59.004 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:08:59.035 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T14:08:59.288 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T14:08:59.335 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T14:09:00.106 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T14:09:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T14:09:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T14:09:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T14:09:00.107 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T14:09:00.448 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T14:09:00.449 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T14:09:00.450 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T14:09:00.450 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T14:09:00.450 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:00.450 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:00.675 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T14:09:01.189 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T14:09:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T14:09:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T14:09:01.190 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T14:09:01.247 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:10 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:01.406 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:01.406 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-25T14:09:01.406 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:09:01.406 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-25T14:09:01.406 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T14:09:01.407 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:01.408 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:01.408 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-25T14:09:01.408 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-03-25T14:09:01.408 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:01.408 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 37 M 2024-03-25T14:09:01.408 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 209 M 2024-03-25T14:09:01.408 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:01.615 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T14:09:01.615 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:01.615 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T14:09:01.616 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T14:09:01.617 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T14:09:01.617 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:01.617 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:09:01.872 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 217 kB/s | 32 kB 00:00 2024-03-25T14:09:01.905 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-03-25T14:09:01.947 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 852 kB/s | 34 kB 00:00 2024-03-25T14:09:01.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-03-25T14:09:01.990 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-25T14:09:02.164 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-25T14:09:02.231 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-25T14:09:02.257 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T14:09:02.540 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-03-25T14:09:02.564 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:21 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:09:02.632 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-25T14:09:02.691 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T14:09:02.714 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:09:02.715 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:09:02.715 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:09:02.715 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:09:02.715 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:09:02.715 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T14:09:02.716 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T14:09:02.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:02.717 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:09:02.717 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:09:02.717 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-03-25T14:09:02.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:02.717 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-03-25T14:09:02.717 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-25T14:09:02.718 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:09:02.791 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-25T14:09:02.901 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 139 kB/s | 18 kB 00:00 2024-03-25T14:09:02.926 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 157 kB/s | 24 kB 00:00 2024-03-25T14:09:02.968 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T14:09:02.994 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T14:09:03.027 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T14:09:03.052 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 734 kB/s | 43 kB 00:00 2024-03-25T14:09:03.078 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 170 kB/s | 52 kB 00:00 2024-03-25T14:09:03.275 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.8 MB/s | 7.4 MB 00:01 2024-03-25T14:09:03.277 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:03.277 INFO:teuthology.orchestra.run.smithi063.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-25T14:09:03.318 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:03.342 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:03.342 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:03.353 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-25T14:09:03.378 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2024-03-25T14:09:03.528 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-25T14:09:03.554 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 426 kB/s | 203 kB 00:00 2024-03-25T14:09:03.662 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-03-25T14:09:03.723 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:03.723 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:03.746 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.6 MB/s | 1.0 MB 00:00 2024-03-25T14:09:03.750 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:03.750 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-25T14:09:03.750 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:09:03.764 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:09:03.764 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:09:03.856 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:09:03.857 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:09:04.383 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:09:04.460 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:04.532 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T14:09:04.562 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T14:09:04.566 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T14:09:04.621 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T14:09:04.661 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:09:04.685 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T14:09:04.699 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T14:09:04.723 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T14:09:04.764 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T14:09:04.857 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T14:09:04.882 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:09:04.914 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:09:04.965 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T14:09:04.992 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T14:09:05.015 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T14:09:05.690 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T14:09:05.739 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T14:09:05.796 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T14:09:05.805 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T14:09:05.829 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T14:09:05.878 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T14:09:05.902 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:09:06.727 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T14:09:06.728 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T14:09:07.096 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T14:09:07.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T14:09:07.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T14:09:07.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T14:09:07.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T14:09:07.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T14:09:07.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T14:09:07.097 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T14:09:07.098 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T14:09:07.098 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:07.098 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:09:07.132 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T14:09:07.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T14:09:07.901 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:26 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:09:08.052 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:09:08.053 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:09:08.053 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:09:08.053 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:09:08.053 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:09:08.053 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T14:09:08.053 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-25T14:09:08.054 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:09:08.055 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.8 M 2024-03-25T14:09:08.055 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-03-25T14:09:08.055 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:09:08.218 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T14:09:08.246 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:09:08.614 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 619 kB/s | 138 kB 00:00 2024-03-25T14:09:08.739 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-25T14:09:09.439 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 237 kB/s | 248 kB 00:01 2024-03-25T14:09:10.690 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T14:09:10.690 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T14:09:10.690 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T14:09:10.690 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T14:09:10.690 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T14:09:10.690 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T14:09:10.690 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T14:09:10.691 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T14:09:10.691 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T14:09:10.691 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T14:09:10.691 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T14:09:10.691 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T14:09:11.159 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T14:09:11.160 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T14:09:11.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T14:09:11.161 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T14:09:11.161 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:11.161 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:11.542 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-03-25T14:09:12.121 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:21 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:12.276 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T14:09:12.278 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-03-25T14:09:12.279 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:12.280 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.0 M 2024-03-25T14:09:12.280 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-03-25T14:09:12.280 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:12.487 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 24 kB 00:00 2024-03-25T14:09:12.512 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2024-03-25T14:09:12.562 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T14:09:12.587 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T14:09:12.613 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T14:09:12.638 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-25T14:09:12.663 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 169 kB/s | 52 kB 00:00 2024-03-25T14:09:12.781 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 15 kB/s | 51 kB 00:03 2024-03-25T14:09:12.897 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-25T14:09:12.922 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-03-25T14:09:13.172 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 398 kB/s | 203 kB 00:00 2024-03-25T14:09:13.256 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 177 kB/s | 59 kB 00:00 2024-03-25T14:09:13.523 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 256 kB/s | 90 kB 00:00 2024-03-25T14:09:13.564 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-25T14:09:13.567 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:13.568 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T14:09:13.568 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:13.584 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:13.585 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:13.687 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:13.689 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:14.216 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:14.388 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T14:09:14.443 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T14:09:14.498 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T14:09:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T14:09:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T14:09:14.675 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T14:09:14.784 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T14:09:14.844 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T14:09:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T14:09:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T14:09:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T14:09:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T14:09:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:09:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T14:09:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T14:09:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T14:09:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T14:09:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T14:09:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T14:09:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T14:09:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T14:09:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T14:09:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T14:09:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T14:09:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T14:09:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T14:09:16.864 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 66 kB/s | 548 kB 00:08 2024-03-25T14:09:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T14:09:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:16.966 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:16.967 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:17.190 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T14:09:17.763 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:26 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:17.920 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:09:17.921 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:09:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:09:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:17.921 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T14:09:17.921 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:17.922 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:09:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-03-25T14:09:17.923 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 6.8 M 2024-03-25T14:09:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 33 M 2024-03-25T14:09:17.923 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:18.474 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-03-25T14:09:18.602 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T14:09:18.624 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 460 kB/s | 138 kB 00:00 2024-03-25T14:09:18.758 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-03-25T14:09:18.783 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 321 kB/s | 51 kB 00:00 2024-03-25T14:09:18.859 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-25T14:09:18.859 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:18.860 INFO:teuthology.orchestra.run.smithi063.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-03-25T14:09:18.873 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:18.883 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:18.883 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:18.977 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:18.978 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:19.285 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:19.464 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T14:09:19.536 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T14:09:20.092 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T14:09:20.174 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T14:09:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T14:09:20.272 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:09:21.087 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:09:21.087 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T14:09:21.087 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T14:09:21.087 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T14:09:21.087 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T14:09:21.087 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:21.397 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:21.649 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-03-25T14:09:22.228 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:31 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:22.384 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:22.385 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:09:22.385 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:22.385 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T14:09:22.385 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:09:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T14:09:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:22.386 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:22.386 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:22.386 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-03-25T14:09:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:22.386 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 964 k 2024-03-25T14:09:22.387 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.8 M 2024-03-25T14:09:22.387 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:22.511 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 632 kB/s | 79 kB 00:00 2024-03-25T14:09:22.686 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 885 kB 00:00 2024-03-25T14:09:22.688 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:22.688 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.1 MB/s | 964 kB 00:00 2024-03-25T14:09:22.688 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:22.709 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:22.709 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:22.785 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:22.785 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:23.117 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T14:09:23.189 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:09:23.790 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:09:23.790 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:09:24.079 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T14:09:24.079 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:24.079 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:24.079 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T14:09:24.079 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:24.079 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:24.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-volume 2024-03-25T14:09:24.931 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:33 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:25.090 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:25.091 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:25.091 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-25T14:09:25.091 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:25.091 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:25.091 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T14:09:25.091 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:09:25.092 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T14:09:25.092 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:25.092 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:25.092 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:25.092 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-03-25T14:09:25.092 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:25.092 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 332 k 2024-03-25T14:09:25.093 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.5 M 2024-03-25T14:09:25.093 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:25.207 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 603 kB/s | 69 kB 00:00 2024-03-25T14:09:25.499 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 650 kB/s | 264 kB 00:00 2024-03-25T14:09:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:Total 814 kB/s | 332 kB 00:00 2024-03-25T14:09:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:25.510 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:25.510 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:25.548 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:25.549 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:25.698 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:25.895 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T14:09:25.935 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:09:25.950 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:09:25.950 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:09:25.950 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T14:09:25.950 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:26.684 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T14:09:27.021 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T14:09:27.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:27.021 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:27.021 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:09:27.021 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T14:09:27.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:27.021 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:27.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-03-25T14:09:27.910 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:36 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:28.066 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:28.067 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:28.067 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 127 k 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 456 k 2024-03-25T14:09:28.068 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:28.404 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 377 kB/s | 127 kB 00:00 2024-03-25T14:09:28.404 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:28.405 INFO:teuthology.orchestra.run.smithi063.stdout:Total 376 kB/s | 127 kB 00:00 2024-03-25T14:09:28.405 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:28.411 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:28.411 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:28.469 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:28.469 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:28.722 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:29.162 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:29.486 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:29.487 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:29.487 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:29.487 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:09:29.487 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:29.487 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:29.679 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-03-25T14:09:30.270 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:39 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:30.370 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:09:30.426 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:30.427 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:09:30.427 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:30.494 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-03-25T14:09:31.077 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:40 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:31.232 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:31.233 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:31.234 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:09:31.234 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:31.234 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 31 k 2024-03-25T14:09:31.234 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 140 k 2024-03-25T14:09:31.234 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:31.406 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 180 kB/s | 31 kB 00:00 2024-03-25T14:09:31.407 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:31.407 INFO:teuthology.orchestra.run.smithi063.stdout:Total 179 kB/s | 31 kB 00:00 2024-03-25T14:09:31.407 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:31.411 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:31.411 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:31.435 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:31.435 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:31.543 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:31.650 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:32.018 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:32.296 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:32.296 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:32.296 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:32.296 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:09:32.296 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:32.296 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:32.455 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-03-25T14:09:33.034 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:42 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:33.134 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:09:33.189 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:33.190 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:09:33.190 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:33.235 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-03-25T14:09:33.812 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:42 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:33.912 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:09:33.967 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:33.968 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:09:33.968 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:34.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-03-25T14:09:34.595 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:43 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:34.694 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:09:34.749 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:34.750 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:09:34.750 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:34.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-03-25T14:09:35.368 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:44 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:35.466 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:09:35.521 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:35.522 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:09:35.522 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:35.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-03-25T14:09:36.143 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:36.242 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:09:36.296 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:36.297 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:09:36.297 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:36.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-03-25T14:09:36.920 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:37.019 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:09:37.076 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:37.077 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:09:37.077 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:37.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-03-25T14:09:37.696 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:46 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:37.852 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:37.853 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:37.853 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:09:37.854 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:37.855 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 85 k 2024-03-25T14:09:37.855 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 231 k 2024-03-25T14:09:37.855 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:38.168 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 272 kB/s | 85 kB 00:00 2024-03-25T14:09:38.168 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:38.168 INFO:teuthology.orchestra.run.smithi063.stdout:Total 271 kB/s | 85 kB 00:00 2024-03-25T14:09:38.169 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:38.176 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:38.176 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:38.233 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:38.234 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:38.384 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:38.485 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:38.856 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:39.153 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:39.153 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:39.153 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:39.153 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:09:39.154 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:39.154 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:39.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-03-25T14:09:39.988 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:40.146 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:40.146 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:40.146 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.1 M 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 13 M 2024-03-25T14:09:40.147 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:40.680 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.8 MB/s | 3.1 MB 00:00 2024-03-25T14:09:40.680 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:40.681 INFO:teuthology.orchestra.run.smithi063.stdout:Total 5.8 MB/s | 3.1 MB 00:00 2024-03-25T14:09:40.681 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:40.690 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:40.690 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:40.766 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:40.766 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:41.215 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:41.297 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:41.322 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:41.322 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:09:41.322 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T14:09:41.322 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:09:41.323 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:09:41.323 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:42.436 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:42.436 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:42.437 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:42.437 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:09:42.437 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:42.437 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:42.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-03-25T14:09:43.253 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:52 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:43.407 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 171 k 2024-03-25T14:09:43.408 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 494 k 2024-03-25T14:09:43.409 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:43.757 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 492 kB/s | 171 kB 00:00 2024-03-25T14:09:43.758 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:43.758 INFO:teuthology.orchestra.run.smithi063.stdout:Total 490 kB/s | 171 kB 00:00 2024-03-25T14:09:43.758 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:43.766 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:43.766 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:43.824 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:43.824 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:43.977 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:09:44.092 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:44.481 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:44.795 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:09:44.795 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:44.795 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:09:44.796 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:09:44.796 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:44.796 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:09:45.008 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:09:45.582 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:09:45.737 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T14:09:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:45.739 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:09:45.739 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:09:45.739 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-03-25T14:09:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:09:45.739 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 91 M 2024-03-25T14:09:45.739 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 498 M 2024-03-25T14:09:45.739 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:09:47.061 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 17 MB/s | 22 MB 00:01 2024-03-25T14:09:50.101 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 16 MB/s | 68 MB 00:04 2024-03-25T14:09:50.103 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:09:50.103 INFO:teuthology.orchestra.run.smithi063.stdout:Total 21 MB/s | 91 MB 00:04 2024-03-25T14:09:50.104 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:09:50.109 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:09:50.109 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:09:50.714 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:09:50.714 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:09:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:10:01.510 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:10:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:10:04.881 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:10:04.881 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T14:10:05.174 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:10:05.174 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:10:05.174 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:10:05.174 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:10:05.174 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:10:05.174 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:10:05.174 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:10:05.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-common-debuginfo 2024-03-25T14:10:05.960 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:10:06.116 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:10:06.117 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:10:06.117 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:10:06.117 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 416 M 2024-03-25T14:10:06.118 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.7 G 2024-03-25T14:10:06.119 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:10:24.014 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 23 MB/s | 416 MB 00:17 2024-03-25T14:10:24.015 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:10:24.015 INFO:teuthology.orchestra.run.smithi063.stdout:Total 23 MB/s | 416 MB 00:17 2024-03-25T14:10:24.015 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:10:24.020 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:10:24.020 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:10:26.610 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:10:26.610 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:11:11.188 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:11:11.296 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:11:19.049 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:11:19.342 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:11:19.342 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:19.342 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:11:19.342 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:19.342 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:19.342 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:11:19.570 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T14:11:20.159 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:11:20.317 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================== 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:11:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:20.319 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 1.4 M 2024-03-25T14:11:20.319 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 6.4 M 2024-03-25T14:11:20.319 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:11:20.485 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-natsort-7.1.1-5.el9.noarch.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/p/python3-natsort-7.1.1-5.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-25T14:11:20.664 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 4.1 MB/s | 1.4 MB 00:00 2024-03-25T14:11:20.664 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:20.664 INFO:teuthology.orchestra.run.smithi063.stdout:Total 4.1 MB/s | 1.4 MB 00:00 2024-03-25T14:11:20.665 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:11:20.668 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:11:20.668 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:11:20.710 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:11:20.710 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:11:20.799 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 447 B/s | 58 kB 02:12 2024-03-25T14:11:20.800 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:20.800 INFO:teuthology.orchestra.run.smithi153.stdout:Total 52 kB/s | 6.8 MB 02:12 2024-03-25T14:11:20.813 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:20.826 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:20.826 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:20.919 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:20.919 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:20.973 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:11:21.082 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:11:21.207 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:21.379 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T14:11:21.433 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T14:11:21.466 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:11:21.746 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:11:21.747 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:21.747 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:11:21.747 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:21.747 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:21.747 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:11:21.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T14:11:21.941 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T14:11:22.018 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T14:11:22.088 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T14:11:22.113 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:11:22.482 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:31 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:11:22.636 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:11:22.637 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:11:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:11:22.637 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:11:22.637 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:11:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T14:11:22.637 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:22.637 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:11:22.638 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:11:22.638 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:11:22.638 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:22.638 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 176 M 2024-03-25T14:11:22.638 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 753 M 2024-03-25T14:11:22.638 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:11:22.882 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T14:11:22.883 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T14:11:22.883 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T14:11:22.883 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T14:11:22.883 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T14:11:22.883 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T14:11:23.208 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T14:11:23.209 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T14:11:23.209 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:23.209 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:23.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-03-25T14:11:24.029 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:43 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:24.179 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:24.179 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-25T14:11:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:24.181 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 964 k 2024-03-25T14:11:24.181 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.8 M 2024-03-25T14:11:24.181 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:24.309 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 616 kB/s | 79 kB 00:00 2024-03-25T14:11:25.143 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 921 kB/s | 885 kB 00:00 2024-03-25T14:11:25.144 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:25.145 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.0 MB/s | 964 kB 00:00 2024-03-25T14:11:25.145 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:25.165 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:25.166 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:25.240 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:25.240 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:25.422 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:25.560 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T14:11:25.625 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:11:26.341 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:11:26.341 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:11:26.639 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T14:11:26.640 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:26.640 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:26.640 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T14:11:26.640 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:26.640 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:26.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2024-03-25T14:11:27.514 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:46 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:27.666 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:27.667 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:27.668 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-25T14:11:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:27.668 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 332 k 2024-03-25T14:11:27.668 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.5 M 2024-03-25T14:11:27.668 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:27.839 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 402 kB/s | 69 kB 00:00 2024-03-25T14:11:28.297 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 419 kB/s | 264 kB 00:00 2024-03-25T14:11:28.298 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:28.299 INFO:teuthology.orchestra.run.smithi153.stdout:Total 526 kB/s | 332 kB 00:00 2024-03-25T14:11:28.299 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:28.307 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:28.308 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:28.348 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:28.348 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:28.500 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:28.688 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T14:11:28.726 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:11:28.740 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T14:11:28.740 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:11:28.741 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T14:11:28.741 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:29.519 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T14:11:29.881 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T14:11:29.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:29.881 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:29.881 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T14:11:29.881 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T14:11:29.882 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:29.882 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:30.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-03-25T14:11:30.416 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 23 MB/s | 176 MB 00:07 2024-03-25T14:11:30.417 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:30.418 INFO:teuthology.orchestra.run.smithi063.stdout:Total 23 MB/s | 176 MB 00:07 2024-03-25T14:11:30.418 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:11:30.422 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:11:30.422 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:11:30.798 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:49 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:30.947 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 127 k 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 456 k 2024-03-25T14:11:30.948 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:31.294 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 366 kB/s | 127 kB 00:00 2024-03-25T14:11:31.295 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:31.295 INFO:teuthology.orchestra.run.smithi153.stdout:Total 365 kB/s | 127 kB 00:00 2024-03-25T14:11:31.295 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:31.301 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:31.301 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:31.355 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:31.356 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:31.502 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:31.547 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:11:31.548 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:11:31.634 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:32.421 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:32.421 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:32.421 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:32.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-03-25T14:11:33.228 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:52 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:33.329 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:11:33.383 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:33.384 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:11:33.384 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:33.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-03-25T14:11:34.020 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:53 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:34.169 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:34.170 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 31 k 2024-03-25T14:11:34.171 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 140 k 2024-03-25T14:11:34.171 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:34.439 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 116 kB/s | 31 kB 00:00 2024-03-25T14:11:34.440 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:34.440 INFO:teuthology.orchestra.run.smithi153.stdout:Total 115 kB/s | 31 kB 00:00 2024-03-25T14:11:34.440 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:34.444 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:34.444 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:34.471 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:34.471 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:34.592 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:34.704 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:35.141 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:35.421 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:35.422 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:35.422 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:35.422 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:35.422 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:35.422 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:35.595 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-03-25T14:11:36.158 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:55 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:36.254 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:11:36.305 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:36.306 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:11:36.306 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:36.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-03-25T14:11:36.930 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:55 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:37.026 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:11:37.078 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:37.079 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:11:37.079 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:37.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-03-25T14:11:37.700 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:56 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:37.797 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:11:37.849 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:37.850 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:11:37.850 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:37.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-03-25T14:11:38.466 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:57 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:38.562 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:11:38.614 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:38.615 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:11:38.615 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:38.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-03-25T14:11:39.240 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:58 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:39.337 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:11:39.389 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:39.390 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:11:39.390 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:39.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-03-25T14:11:40.006 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:59 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:40.104 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:11:40.155 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:40.156 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:11:40.156 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:40.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-03-25T14:11:40.774 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:59 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:40.921 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:40.921 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:11:40.922 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:40.923 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 85 k 2024-03-25T14:11:40.923 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 231 k 2024-03-25T14:11:40.923 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:41.260 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 254 kB/s | 85 kB 00:00 2024-03-25T14:11:41.261 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:41.261 INFO:teuthology.orchestra.run.smithi153.stdout:Total 253 kB/s | 85 kB 00:00 2024-03-25T14:11:41.261 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:41.267 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:41.268 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:41.323 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:41.323 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:41.456 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:41.574 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:41.991 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:42.303 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:42.303 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:42.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-03-25T14:11:43.132 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:04:02 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:43.282 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:43.282 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.1 M 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 13 M 2024-03-25T14:11:43.283 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:43.670 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 8.0 MB/s | 3.1 MB 00:00 2024-03-25T14:11:43.671 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:43.671 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-03-25T14:11:43.671 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:43.682 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:43.682 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:43.755 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:43.755 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:44.180 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:44.270 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:44.270 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T14:11:44.270 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T14:11:44.271 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:11:44.271 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T14:11:44.271 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:45.519 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:45.520 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:45.520 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:45.520 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:45.520 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:45.520 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:45.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-03-25T14:11:46.350 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:46.498 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:46.498 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:46.498 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-25T14:11:46.498 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 171 k 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 494 k 2024-03-25T14:11:46.499 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:46.715 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 794 kB/s | 171 kB 00:00 2024-03-25T14:11:46.716 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:11:46.716 INFO:teuthology.orchestra.run.smithi153.stdout:Total 788 kB/s | 171 kB 00:00 2024-03-25T14:11:46.716 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:11:46.724 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:11:46.724 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:11:46.780 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:11:46.780 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:11:46.926 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:11:47.039 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:47.478 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:47.778 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:11:47.778 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:47.778 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:11:47.778 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:47.778 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:47.778 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:11:48.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:11:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:04:07 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:11:48.753 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T14:11:48.754 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:48.755 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:11:48.755 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:11:48.755 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-25T14:11:48.755 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:11:48.755 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 91 M 2024-03-25T14:11:48.755 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 498 M 2024-03-25T14:11:48.756 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:11:50.602 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:11:50.700 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:11:55.245 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:11:55.532 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:11:55.532 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:55.532 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:11:55.532 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:11:55.532 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:55.533 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:11:55.732 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test-debuginfo 2024-03-25T14:11:56.301 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:11:56.456 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:11:56.457 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:11:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:11:56.457 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:11:56.457 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:11:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T14:11:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:56.457 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:11:56.458 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:11:56.458 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:11:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:11:56.458 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 672 M 2024-03-25T14:11:56.458 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 3.1 G 2024-03-25T14:11:56.458 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:11:58.234 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 2.4 MB/s | 22 MB 00:09 2024-03-25T14:12:05.716 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 4.0 MB/s | 68 MB 00:16 2024-03-25T14:12:05.718 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:12:05.718 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.3 MB/s | 91 MB 00:16 2024-03-25T14:12:05.719 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:12:05.724 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:12:05.724 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:12:06.318 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:12:06.319 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:12:09.412 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:12:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T14:12:16.617 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:12:19.949 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T14:12:19.950 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T14:12:20.240 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T14:12:20.240 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:12:20.240 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:12:20.240 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:12:20.240 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:12:20.240 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:12:20.240 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:12:20.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-common-debuginfo 2024-03-25T14:12:21.084 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:12:21.240 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:12:21.240 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 416 M 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.7 G 2024-03-25T14:12:21.241 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:12:34.224 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 18 MB/s | 672 MB 00:37 2024-03-25T14:12:34.225 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:12:34.225 INFO:teuthology.orchestra.run.smithi063.stdout:Total 18 MB/s | 672 MB 00:37 2024-03-25T14:12:34.225 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:12:34.231 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:12:34.231 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:12:38.385 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:12:38.386 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:12:48.444 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 15 MB/s | 416 MB 00:27 2024-03-25T14:12:48.445 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:12:48.445 INFO:teuthology.orchestra.run.smithi153.stdout:Total 15 MB/s | 416 MB 00:27 2024-03-25T14:12:48.445 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:12:48.449 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:12:48.449 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:12:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:12:50.915 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:13:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:13:33.280 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:13:41.095 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:13:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:13:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:41.350 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:13:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:13:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:41.350 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:13:41.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T14:13:42.162 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:01 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:13:42.310 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:13:42.311 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:13:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:13:42.311 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:13:42.311 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:13:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T14:13:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:42.312 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:13:42.312 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T14:13:42.312 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:13:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:42.312 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.4 M 2024-03-25T14:13:42.312 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 6.4 M 2024-03-25T14:13:42.312 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:13:42.916 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 2.4 MB/s | 1.4 MB 00:00 2024-03-25T14:13:42.917 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:13:42.917 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.4 MB/s | 1.4 MB 00:00 2024-03-25T14:13:42.917 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:13:42.921 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:13:42.921 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:13:42.963 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:13:42.963 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:13:43.221 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:13:43.324 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:13:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:13:44.036 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T14:13:44.037 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:44.037 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:13:44.037 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:13:44.037 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:44.037 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:13:44.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T14:13:44.772 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:13:44.921 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:13:44.922 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:13:44.922 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:13:44.922 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:13:44.922 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:13:44.922 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T14:13:44.922 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:44.922 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:13:44.923 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:13:44.923 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:13:44.923 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:13:44.923 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 176 M 2024-03-25T14:13:44.923 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 753 M 2024-03-25T14:13:44.923 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:13:53.535 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:13:53.635 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 20 MB/s | 176 MB 00:08 2024-03-25T14:13:53.636 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:13:53.637 INFO:teuthology.orchestra.run.smithi153.stdout:Total 20 MB/s | 176 MB 00:08 2024-03-25T14:13:53.637 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:13:53.641 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:13:53.641 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:13:53.653 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:13:54.706 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:13:54.707 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:14:01.001 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:14:01.273 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:14:01.658 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:01.658 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:14:01.658 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:14:01.658 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:01.658 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:14:01.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:14:02.236 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:14:02.335 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:14:02.389 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:14:02.390 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-03-25T14:14:02.391 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:14:02.450 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T14:14:03.023 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:14:03.176 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:14:03.177 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:03.177 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:14:03.177 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:03.177 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 20 M 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 103 M 2024-03-25T14:14:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:14:04.671 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 13 MB/s | 20 MB 00:01 2024-03-25T14:14:04.671 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:14:04.672 INFO:teuthology.orchestra.run.smithi063.stdout:Total 13 MB/s | 20 MB 00:01 2024-03-25T14:14:04.672 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:14:04.676 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:14:04.676 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:14:04.834 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:14:04.834 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:14:07.194 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:14:07.297 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:14:08.225 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:14:08.505 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:14:08.505 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:08.505 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:14:08.506 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:14:08.506 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:08.506 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:14:08.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T14:14:09.242 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:06:18 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:14:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:14:09.396 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 48 M 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 173 M 2024-03-25T14:14:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:14:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 17 MB/s | 48 MB 00:02 2024-03-25T14:14:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:14:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-25T14:14:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:14:12.170 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:14:12.171 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:14:12.499 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:14:12.499 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:14:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:14:12.765 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:14:17.293 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:14:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:14:17.552 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T14:14:17.552 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:14:17.552 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:14:17.552 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:14:17.552 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:14:17.552 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:14:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:14:17.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test-debuginfo 2024-03-25T14:14:18.305 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:37 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:14:18.453 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:14:18.454 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:14:18.455 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:14:18.455 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 672 M 2024-03-25T14:14:18.455 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 3.1 G 2024-03-25T14:14:18.455 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:14:18.948 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:14:19.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:14:19.219 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:19.219 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:14:19.219 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:14:19.219 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:19.219 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:14:19.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T14:14:19.940 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:06:28 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:14:20.095 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:14:20.096 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:20.096 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:14:20.096 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:20.096 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:14:20.096 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T14:14:20.096 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:20.096 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:14:20.097 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:14:20.097 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:14:20.097 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:14:20.097 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 99 M 2024-03-25T14:14:20.097 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 449 M 2024-03-25T14:14:20.097 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:14:48.273 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 3.5 MB/s | 99 MB 00:28 2024-03-25T14:14:48.274 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:14:48.274 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.5 MB/s | 99 MB 00:28 2024-03-25T14:14:48.275 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:14:48.278 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:14:48.278 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:14:48.919 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:14:48.919 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:14:49.899 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 21 MB/s | 672 MB 00:31 2024-03-25T14:14:49.900 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:14:49.900 INFO:teuthology.orchestra.run.smithi153.stdout:Total 21 MB/s | 672 MB 00:31 2024-03-25T14:14:49.901 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:14:49.906 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:14:49.906 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:14:53.866 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:14:53.866 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:14:59.782 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:14:59.949 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:15:02.715 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:15:03.093 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:15:03.093 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:03.093 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:15:03.093 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:15:03.093 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:03.093 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:15:03.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T14:15:03.837 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:07:12 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:15:03.993 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:15:03.993 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:15:03.993 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:15:03.993 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 287 M 2024-03-25T14:15:03.994 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.2 G 2024-03-25T14:15:03.995 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:15:16.400 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 23 MB/s | 287 MB 00:12 2024-03-25T14:15:16.400 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:15:16.401 INFO:teuthology.orchestra.run.smithi063.stdout:Total 23 MB/s | 287 MB 00:12 2024-03-25T14:15:16.401 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:15:16.404 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:15:16.405 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:15:18.196 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:15:18.196 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:15:49.360 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:15:49.581 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:15:56.751 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:15:57.053 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:15:57.053 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:57.053 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:15:57.054 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:15:57.054 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:57.054 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:15:57.345 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T14:15:57.916 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:06 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:15:58.071 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:15:58.072 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:15:58.072 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:15:58.072 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:15:58.072 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:15:58.072 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T14:15:58.072 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:58.073 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:15:58.073 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:15:58.073 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:15:58.073 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:15:58.073 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 5.9 M 2024-03-25T14:15:58.073 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 28 M 2024-03-25T14:15:58.073 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:15:58.732 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 9.0 MB/s | 5.9 MB 00:00 2024-03-25T14:15:58.733 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:15:58.733 INFO:teuthology.orchestra.run.smithi063.stdout:Total 9.0 MB/s | 5.9 MB 00:00 2024-03-25T14:15:58.733 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:15:58.737 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:15:58.737 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:15:58.809 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:15:58.809 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:15:59.578 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:15:59.693 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:00.242 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:00.528 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:00.529 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:00.695 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel-debuginfo 2024-03-25T14:16:01.272 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:10 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:01.429 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:01.430 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:01.430 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:01.430 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:01.430 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 109 k 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 440 k 2024-03-25T14:16:01.431 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:01.644 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 511 kB/s | 109 kB 00:00 2024-03-25T14:16:01.645 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:01.645 INFO:teuthology.orchestra.run.smithi063.stdout:Total 507 kB/s | 109 kB 00:00 2024-03-25T14:16:01.645 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:01.648 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:01.648 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:01.684 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:01.684 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:01.808 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:01.913 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:16:02.289 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:16:02.576 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:16:02.576 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:02.576 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:02.576 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:02.576 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:02.576 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:02.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T14:16:03.320 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:12 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:03.474 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:03.475 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:03.475 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:03.475 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T14:16:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:03.476 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:03.476 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:03.476 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:03.476 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 5.4 M 2024-03-25T14:16:03.476 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 25 M 2024-03-25T14:16:03.476 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:04.658 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:16:05.311 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:05.888 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 2.2 MB/s | 5.4 MB 00:02 2024-03-25T14:16:05.889 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:05.890 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.2 MB/s | 5.4 MB 00:02 2024-03-25T14:16:05.890 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:05.894 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:05.894 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:05.965 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:05.966 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:06.668 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:06.782 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:16:07.284 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:16:07.566 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:16:07.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:07.567 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:07.567 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:07.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:07.567 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:07.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2-debuginfo 2024-03-25T14:16:08.361 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:17 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:08.515 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:08.515 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 44 M 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 253 M 2024-03-25T14:16:08.516 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:10.874 INFO:teuthology.orchestra.run.smithi063.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 19 MB/s | 44 MB 00:02 2024-03-25T14:16:10.875 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:10.875 INFO:teuthology.orchestra.run.smithi063.stdout:Total 19 MB/s | 44 MB 00:02 2024-03-25T14:16:10.876 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:10.880 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:10.880 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:11.194 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:11.194 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:13.523 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:13.811 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:13.812 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:13.812 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:16:13.812 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:13.812 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:13.812 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:16:14.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base-debuginfo 2024-03-25T14:16:14.684 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:33 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:16:14.780 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T14:16:14.832 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:16:14.833 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T14:16:14.833 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:16:14.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T14:16:15.466 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:34 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:16:15.617 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:16:15.617 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:15.617 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 20 M 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 103 M 2024-03-25T14:16:15.618 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:16:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:16.711 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 18 MB/s | 20 MB 00:01 2024-03-25T14:16:16.712 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:16.712 INFO:teuthology.orchestra.run.smithi153.stdout:Total 18 MB/s | 20 MB 00:01 2024-03-25T14:16:16.712 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:16:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:16:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:16:16.864 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:16:16.865 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:16:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:18.716 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:16:18.716 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:18.717 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:18.717 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:18.717 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:18.717 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:18.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1-debuginfo 2024-03-25T14:16:19.092 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:16:19.184 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:19.458 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:28 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:19.614 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:19.615 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:19.615 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:19.615 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 52 M 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 308 M 2024-03-25T14:16:19.616 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:20.186 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:20.468 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:20.468 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:20.468 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:16:20.468 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:20.469 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:20.469 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:16:20.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T14:16:21.191 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:40 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:16:21.339 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:16:21.339 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 48 M 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 173 M 2024-03-25T14:16:21.340 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:16:24.388 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 16 MB/s | 48 MB 00:03 2024-03-25T14:16:24.389 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:24.389 INFO:teuthology.orchestra.run.smithi153.stdout:Total 16 MB/s | 48 MB 00:03 2024-03-25T14:16:24.390 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:16:24.394 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:16:24.394 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:16:24.708 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:16:24.708 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:16:25.235 INFO:teuthology.orchestra.run.smithi063.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 9.2 MB/s | 52 MB 00:05 2024-03-25T14:16:25.235 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:25.236 INFO:teuthology.orchestra.run.smithi063.stdout:Total 9.2 MB/s | 52 MB 00:05 2024-03-25T14:16:25.236 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:25.240 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:25.240 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:25.600 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:25.600 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:29.441 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:16:29.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:30.927 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:31.201 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:31.201 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:31.201 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:16:31.201 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:31.201 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:31.202 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:16:31.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T14:16:31.924 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:50 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:16:31.951 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:32.054 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:16:32.075 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:16:32.075 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:32.075 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:16:32.075 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:32.075 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:16:32.076 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T14:16:32.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:32.076 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:16:32.076 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:32.076 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:16:32.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:32.076 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 99 M 2024-03-25T14:16:32.077 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 449 M 2024-03-25T14:16:32.077 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:16:34.165 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:16:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:16:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:34.446 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:34.446 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:34.618 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T14:16:35.188 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:44 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:35.343 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:35.344 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:35.345 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:35.345 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 615 k 2024-03-25T14:16:35.345 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.1 M 2024-03-25T14:16:35.345 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:35.884 INFO:teuthology.orchestra.run.smithi063.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 1.1 MB/s | 615 kB 00:00 2024-03-25T14:16:35.885 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:35.885 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.1 MB/s | 615 kB 00:00 2024-03-25T14:16:35.885 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:35.886 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:35.886 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:35.923 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:35.924 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:36.223 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:16:36.627 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:16:36.913 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:16:36.913 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:36.913 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:36.913 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:36.913 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:36.913 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:37.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados-debuginfo 2024-03-25T14:16:37.657 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:46 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:37.813 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:37.814 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 771 k 2024-03-25T14:16:37.815 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.7 M 2024-03-25T14:16:37.815 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:38.132 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 2.4 MB/s | 771 kB 00:00 2024-03-25T14:16:38.133 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:38.133 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.4 MB/s | 771 kB 00:00 2024-03-25T14:16:38.133 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:38.136 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:38.137 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:38.175 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:38.176 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:38.363 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:38.459 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:16:38.756 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 15 MB/s | 99 MB 00:06 2024-03-25T14:16:38.757 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:38.758 INFO:teuthology.orchestra.run.smithi153.stdout:Total 15 MB/s | 99 MB 00:06 2024-03-25T14:16:38.758 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:16:38.762 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:16:38.762 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:16:38.833 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:16:39.104 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:16:39.104 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:39.104 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:39.104 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:39.104 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:39.105 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:39.308 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T14:16:39.372 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:16:39.373 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:16:39.883 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:48 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:40.041 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 1.1 M 2024-03-25T14:16:40.042 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 4.3 M 2024-03-25T14:16:40.043 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:40.485 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 2.5 MB/s | 1.1 MB 00:00 2024-03-25T14:16:40.486 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:40.486 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.5 MB/s | 1.1 MB 00:00 2024-03-25T14:16:40.486 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:40.489 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:40.490 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:40.530 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:40.530 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:40.762 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:40.870 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:16:41.261 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:16:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:16:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:41.537 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:41.538 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:41.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T14:16:42.303 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:51 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:42.459 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 337 k 2024-03-25T14:16:42.460 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.0 M 2024-03-25T14:16:42.461 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:42.696 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 1.4 MB/s | 337 kB 00:00 2024-03-25T14:16:42.697 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:42.697 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.4 MB/s | 337 kB 00:00 2024-03-25T14:16:42.697 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:42.701 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:42.701 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:42.737 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:42.737 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:42.875 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:42.973 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:16:43.340 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:16:43.618 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:16:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:43.619 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:43.619 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:43.619 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:43.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T14:16:44.364 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:53 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:44.519 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:44.520 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:44.521 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 509 k 2024-03-25T14:16:44.521 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.5 M 2024-03-25T14:16:44.521 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:44.916 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 509 kB 00:00 2024-03-25T14:16:44.917 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:44.917 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.3 MB/s | 509 kB 00:00 2024-03-25T14:16:44.917 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:44.920 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:44.920 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:44.958 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:44.959 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:45.123 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:45.235 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:45.624 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:45.903 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:45.903 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:45.903 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:16:45.903 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:45.903 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:45.903 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:16:46.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T14:16:46.653 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:08:55 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:16:46.809 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:16:46.810 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:46.810 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:16:46.810 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:46.810 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:16:46.810 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 52 M 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 298 M 2024-03-25T14:16:46.811 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:16:49.639 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:16:49.736 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:51.452 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 11 MB/s | 52 MB 00:04 2024-03-25T14:16:51.453 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:16:51.453 INFO:teuthology.orchestra.run.smithi063.stdout:Total 11 MB/s | 52 MB 00:04 2024-03-25T14:16:51.454 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:16:51.458 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:16:51.458 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:16:51.809 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:16:51.810 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:16:52.666 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:52.944 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:16:52.945 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:52.945 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:16:52.945 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:16:52.945 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:52.945 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:16:53.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T14:16:53.686 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:09:12 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:16:53.835 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:16:53.836 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:53.836 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:16:53.836 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:53.836 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:16:53.836 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 287 M 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.2 G 2024-03-25T14:16:53.837 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:16:58.055 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:16:58.171 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:17:00.303 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:17:00.576 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:17:00.576 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:17:00.577 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:17:00.577 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:00.577 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:17:00.577 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:17:00.741 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T14:17:01.318 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:09:10 ago on Mon 25 Mar 2024 02:07:51 PM UTC. 2024-03-25T14:17:01.473 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-25T14:17:01.473 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:17:01.473 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 1.3 M 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 5.5 M 2024-03-25T14:17:01.474 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-25T14:17:02.154 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.8 MB/s | 1.3 MB 00:00 2024-03-25T14:17:02.155 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:02.155 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.8 MB/s | 1.3 MB 00:00 2024-03-25T14:17:02.155 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-25T14:17:02.159 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-25T14:17:02.159 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-25T14:17:02.202 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-25T14:17:02.202 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-25T14:17:02.478 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-25T14:17:02.582 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:17:02.970 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:17:03.234 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:17:03.234 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:17:03.234 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-25T14:17:03.234 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:03.234 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:17:03.234 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-25T14:17:03.407 DEBUG:teuthology.parallel:result is None 2024-03-25T14:17:07.415 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 21 MB/s | 287 MB 00:13 2024-03-25T14:17:07.416 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:07.416 INFO:teuthology.orchestra.run.smithi153.stdout:Total 21 MB/s | 287 MB 00:13 2024-03-25T14:17:07.417 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:17:07.421 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:17:07.421 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:17:09.125 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:17:09.126 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:17:38.341 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:17:38.435 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:17:45.800 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:17:46.097 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:17:46.097 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:46.097 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:17:46.097 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:46.097 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:46.097 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:17:46.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T14:17:46.862 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:05 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:17:47.011 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:17:47.012 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:47.012 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:17:47.012 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 5.9 M 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 28 M 2024-03-25T14:17:47.013 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:17:47.739 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 8.2 MB/s | 5.9 MB 00:00 2024-03-25T14:17:47.740 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:47.740 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.2 MB/s | 5.9 MB 00:00 2024-03-25T14:17:47.740 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:17:47.744 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:17:47.744 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:17:47.812 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:17:47.812 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:17:48.535 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:17:48.637 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:17:49.204 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:17:49.479 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:17:49.480 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:49.480 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:17:49.480 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:49.480 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:49.480 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:17:49.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel-debuginfo 2024-03-25T14:17:50.239 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:09 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:17:50.387 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:17:50.388 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:50.388 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:17:50.388 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:50.388 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:17:50.388 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T14:17:50.388 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:50.389 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:17:50.389 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:50.389 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:17:50.389 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:50.389 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 109 k 2024-03-25T14:17:50.389 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 440 k 2024-03-25T14:17:50.389 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:17:50.592 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 535 kB/s | 109 kB 00:00 2024-03-25T14:17:50.592 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:50.593 INFO:teuthology.orchestra.run.smithi153.stdout:Total 531 kB/s | 109 kB 00:00 2024-03-25T14:17:50.593 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:17:50.596 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:17:50.596 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:17:50.630 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:17:50.630 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:17:50.749 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:17:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:17:51.240 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:17:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:17:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:51.527 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:17:51.527 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:51.528 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:51.528 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:17:51.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T14:17:52.270 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:11 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:17:52.420 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:17:52.421 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:52.421 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:17:52.421 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:52.422 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:17:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T14:17:52.422 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:52.422 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:17:52.422 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:52.422 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:17:52.423 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:52.423 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 5.4 M 2024-03-25T14:17:52.423 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 25 M 2024-03-25T14:17:52.423 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:17:53.095 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 7.9 MB/s | 5.4 MB 00:00 2024-03-25T14:17:53.096 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:53.096 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.9 MB/s | 5.4 MB 00:00 2024-03-25T14:17:53.096 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:17:53.100 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:17:53.100 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:17:53.165 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:17:53.166 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:17:53.828 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:17:53.927 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:17:54.486 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:17:54.771 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:17:54.772 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:54.772 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:17:54.772 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:17:54.772 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:54.772 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:17:54.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2-debuginfo 2024-03-25T14:17:55.533 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:14 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:17:55.683 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:17:55.684 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:55.684 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:17:55.684 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:55.684 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:17:55.684 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T14:17:55.684 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:55.684 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:17:55.685 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:17:55.685 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:17:55.685 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:17:55.685 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 44 M 2024-03-25T14:17:55.685 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 253 M 2024-03-25T14:17:55.685 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:17:57.840 INFO:teuthology.orchestra.run.smithi153.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 21 MB/s | 44 MB 00:02 2024-03-25T14:17:57.841 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:17:57.841 INFO:teuthology.orchestra.run.smithi153.stdout:Total 21 MB/s | 44 MB 00:02 2024-03-25T14:17:57.841 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:17:57.846 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:17:57.846 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:17:58.145 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:17:58.145 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:03.234 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:03.330 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:18:05.199 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:18:05.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T14:18:05.483 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:05.484 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:05.484 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:05.484 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:05.484 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:05.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1-debuginfo 2024-03-25T14:18:06.226 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:25 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:06.377 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:06.377 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 52 M 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 308 M 2024-03-25T14:18:06.378 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:08.909 INFO:teuthology.orchestra.run.smithi153.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 20 MB/s | 52 MB 00:02 2024-03-25T14:18:08.909 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:08.910 INFO:teuthology.orchestra.run.smithi153.stdout:Total 20 MB/s | 52 MB 00:02 2024-03-25T14:18:08.910 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:08.914 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:08.914 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:09.258 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:09.258 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:15.246 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:15.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:17.469 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:17.758 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:17.758 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:17.758 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:17.758 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:17.758 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:17.758 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:17.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T14:18:18.525 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:37 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:18.674 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:18.675 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 615 k 2024-03-25T14:18:18.676 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.1 M 2024-03-25T14:18:18.676 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:18.928 INFO:teuthology.orchestra.run.smithi153.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 2.4 MB/s | 615 kB 00:00 2024-03-25T14:18:18.929 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:18.929 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.4 MB/s | 615 kB 00:00 2024-03-25T14:18:18.929 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:18.932 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:18.932 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:18.970 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:18.970 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:19.137 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:19.244 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:18:19.647 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:18:19.932 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T14:18:19.932 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:19.932 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:19.932 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:19.932 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:19.933 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:20.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados-debuginfo 2024-03-25T14:18:20.697 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:39 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:20.848 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:20.849 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:20.850 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 771 k 2024-03-25T14:18:20.850 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.7 M 2024-03-25T14:18:20.850 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:21.506 INFO:teuthology.orchestra.run.smithi153.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 1.1 MB/s | 771 kB 00:00 2024-03-25T14:18:21.507 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:21.507 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.1 MB/s | 771 kB 00:00 2024-03-25T14:18:21.507 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:21.510 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:21.510 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:21.549 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:21.549 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:21.730 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:21.838 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:18:22.233 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:18:22.505 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T14:18:22.506 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:22.506 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:22.506 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:22.506 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:22.506 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:22.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T14:18:23.275 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:42 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:23.425 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:23.426 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:23.426 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:23.426 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.1 M 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 4.3 M 2024-03-25T14:18:23.427 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:24.059 INFO:teuthology.orchestra.run.smithi153.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 1.8 MB/s | 1.1 MB 00:00 2024-03-25T14:18:24.060 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:24.060 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.8 MB/s | 1.1 MB 00:00 2024-03-25T14:18:24.060 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:24.064 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:24.064 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:24.103 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:24.103 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:24.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:18:24.835 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:18:25.114 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:18:25.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:25.114 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:25.114 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:25.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:25.114 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:25.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T14:18:25.869 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:44 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:26.019 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:26.020 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:26.020 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:26.020 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:26.020 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:26.020 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T14:18:26.020 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:26.020 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:26.021 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:26.021 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:26.021 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:26.021 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 337 k 2024-03-25T14:18:26.021 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.0 M 2024-03-25T14:18:26.021 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:26.288 INFO:teuthology.orchestra.run.smithi153.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 1.2 MB/s | 337 kB 00:00 2024-03-25T14:18:26.289 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:26.289 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 337 kB 00:00 2024-03-25T14:18:26.289 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:26.292 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:26.292 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:26.326 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:26.327 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:26.467 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:26.570 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:18:26.948 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:18:27.232 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T14:18:27.232 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:27.232 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:27.233 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:27.233 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:27.233 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:27.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T14:18:27.985 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:46 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:28.134 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:28.135 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:28.135 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:28.135 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:28.135 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 509 k 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.5 M 2024-03-25T14:18:28.136 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:28.607 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 509 kB 00:00 2024-03-25T14:18:28.608 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:28.608 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.1 MB/s | 509 kB 00:00 2024-03-25T14:18:28.608 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:28.611 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:28.612 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:28.647 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:28.648 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:28.806 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:28.906 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:18:29.297 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:18:29.583 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T14:18:29.583 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:29.583 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:29.583 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:29.583 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:29.583 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:29.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T14:18:30.348 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:10:49 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:30.497 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 52 M 2024-03-25T14:18:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 298 M 2024-03-25T14:18:30.499 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:33.302 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 18 MB/s | 52 MB 00:02 2024-03-25T14:18:33.303 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:33.303 INFO:teuthology.orchestra.run.smithi153.stdout:Total 18 MB/s | 52 MB 00:02 2024-03-25T14:18:33.304 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:33.308 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:33.308 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:33.642 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:33.643 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:39.557 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:18:41.703 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:18:41.988 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T14:18:41.988 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:41.988 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:41.988 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:41.988 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:41.988 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:42.166 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T14:18:42.731 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:11:01 ago on Mon 25 Mar 2024 02:07:41 PM UTC. 2024-03-25T14:18:42.879 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T14:18:42.880 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:42.880 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T14:18:42.880 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:42.880 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T14:18:42.880 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T14:18:42.880 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:42.881 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T14:18:42.881 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T14:18:42.881 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T14:18:42.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:42.881 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.3 M 2024-03-25T14:18:42.881 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 5.5 M 2024-03-25T14:18:42.881 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T14:18:43.762 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.4 MB/s | 1.3 MB 00:00 2024-03-25T14:18:43.763 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T14:18:43.763 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.4 MB/s | 1.3 MB 00:00 2024-03-25T14:18:43.763 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T14:18:43.766 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T14:18:43.766 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T14:18:43.807 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T14:18:43.808 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T14:18:44.038 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T14:18:44.145 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:44.544 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:44.829 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T14:18:44.830 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:44.830 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T14:18:44.830 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T14:18:44.830 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:44.830 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T14:18:45.013 DEBUG:teuthology.parallel:result is None 2024-03-25T14:18:45.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:18:45.246 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T14:18:45.271 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T14:18:45.272 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T14:18:45.272 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T14:18:45.274 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T14:18:45.492 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T14:18:45.515 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T14:18:45.515 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T14:18:45.516 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T14:18:45.517 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T14:18:45.518 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:45.518 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T14:18:45.554 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:18:45.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T14:18:45.584 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T14:18:45.584 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:45.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T14:18:45.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T14:18:45.695 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:18:45.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T14:18:45.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T14:18:45.794 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T14:18:45.794 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:45.794 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T14:18:45.826 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T14:18:45.894 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:18:45.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T14:18:45.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T14:18:45.989 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T14:18:45.990 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:45.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T14:18:46.017 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T14:18:46.085 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:18:46.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T14:18:46.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T14:18:46.180 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T14:18:46.364 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T14:18:46.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2024-03-25T14:18:46.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-03-25T14:18:46.395 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T14:18:46.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T14:18:46.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T14:18:46.463 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T14:18:46.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T14:18:46.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T14:18:46.567 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T14:18:46.568 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T14:18:46.568 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621266', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8uAacY53DXOKDITja8/1iob9sZhkTWetwQMGMhL6oONG+hzEmsexaheGDGSltZaL6A1lH72N2TmBFw8y5A9x0=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtg8u7uZrI0hoPbbZj5mH/sRNnx6SWjde0GamNdoFafJf5+bKIYtjqXT3MG6SDQuE/1xZlt4pWKIyEHtov3+RI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T14:18:46.568 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T14:18:46.571 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T14:18:46.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T14:18:46.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T14:18:46.653 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:46.654 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T14:18:46.705 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T14:18:46.705 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-03-25T14:18:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T14:18:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:46.759 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T14:18:46.759 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:46.759 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:46.759 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 14:09:41.583476672 +0000 2024-03-25T14:18:46.759 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 14:06:46.814436160 +0000 2024-03-25T14:18:46.760 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 14:06:46.814436160 +0000 2024-03-25T14:18:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 14:06:46.814436160 +0000 2024-03-25T14:18:46.760 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T14:18:46.823 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T14:18:46.824 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T14:18:46.824 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000237537 s, 2.2 MB/s 2024-03-25T14:18:46.825 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T14:18:46.882 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-03-25T14:18:46.936 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T14:18:46.936 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:46.936 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T14:18:46.936 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:46.936 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:46.937 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 14:09:41.584476661 +0000 2024-03-25T14:18:46.937 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 14:06:47.157432579 +0000 2024-03-25T14:18:46.937 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 14:06:47.157432579 +0000 2024-03-25T14:18:46.937 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 14:06:47.157432579 +0000 2024-03-25T14:18:46.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T14:18:47.002 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T14:18:47.002 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T14:18:47.002 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000199451 s, 2.6 MB/s 2024-03-25T14:18:47.003 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T14:18:47.059 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-03-25T14:18:47.114 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T14:18:47.114 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:47.114 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T14:18:47.115 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:47.115 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:47.115 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 14:09:41.584476661 +0000 2024-03-25T14:18:47.115 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 14:06:47.504428955 +0000 2024-03-25T14:18:47.115 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 14:06:47.504428955 +0000 2024-03-25T14:18:47.115 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 14:06:47.504428955 +0000 2024-03-25T14:18:47.115 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T14:18:47.179 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T14:18:47.180 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T14:18:47.180 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000185425 s, 2.8 MB/s 2024-03-25T14:18:47.181 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T14:18:47.236 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-03-25T14:18:47.291 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T14:18:47.291 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:47.291 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T14:18:47.291 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:47.291 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:47.291 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-25 14:09:41.585476651 +0000 2024-03-25T14:18:47.292 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-25 14:06:47.831425540 +0000 2024-03-25T14:18:47.292 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-25 14:06:47.831425540 +0000 2024-03-25T14:18:47.292 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-25 14:06:47.831425540 +0000 2024-03-25T14:18:47.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T14:18:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-25T14:18:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-25T14:18:47.356 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000163633 s, 3.1 MB/s 2024-03-25T14:18:47.357 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T14:18:47.412 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:18:47.412 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:18:47.413 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T14:18:47.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T14:18:47.430 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-25T14:18:47.487 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T14:18:47.487 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T14:18:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 14:11:44.532853487 +0000 2024-03-25T14:18:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 14:06:44.683251999 +0000 2024-03-25T14:18:47.488 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 14:06:44.683251999 +0000 2024-03-25T14:18:47.488 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 14:06:44.683251999 +0000 2024-03-25T14:18:47.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T14:18:47.555 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T14:18:47.555 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T14:18:47.555 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000213947 s, 2.4 MB/s 2024-03-25T14:18:47.556 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T14:18:47.614 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-25T14:18:47.670 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T14:18:47.670 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:47.670 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T14:18:47.671 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:47.671 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:47.671 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 14:11:44.533853467 +0000 2024-03-25T14:18:47.671 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 14:06:45.039244875 +0000 2024-03-25T14:18:47.671 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 14:06:45.039244875 +0000 2024-03-25T14:18:47.671 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 14:06:45.039244875 +0000 2024-03-25T14:18:47.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T14:18:47.736 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T14:18:47.736 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T14:18:47.737 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000218232 s, 2.3 MB/s 2024-03-25T14:18:47.738 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T14:18:47.794 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-25T14:18:47.850 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 14:11:44.533853467 +0000 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 14:06:45.377238112 +0000 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 14:06:45.377238112 +0000 2024-03-25T14:18:47.851 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 14:06:45.377238112 +0000 2024-03-25T14:18:47.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T14:18:47.917 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T14:18:47.917 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T14:18:47.917 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000215244 s, 2.4 MB/s 2024-03-25T14:18:47.919 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T14:18:47.975 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-25T14:18:48.030 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T14:18:48.030 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T14:18:48.030 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T14:18:48.030 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T14:18:48.030 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T14:18:48.031 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 14:11:44.533853467 +0000 2024-03-25T14:18:48.031 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 14:06:45.731231028 +0000 2024-03-25T14:18:48.031 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 14:06:45.731231028 +0000 2024-03-25T14:18:48.031 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 14:06:45.731231028 +0000 2024-03-25T14:18:48.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T14:18:48.095 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T14:18:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T14:18:48.096 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000190889 s, 2.7 MB/s 2024-03-25T14:18:48.097 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T14:18:48.152 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:18:48.152 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T14:18:48.153 INFO:tasks.ceph:Generating config... 2024-03-25T14:18:48.172 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T14:18:48.173 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T14:18:48.173 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T14:18:48.173 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T14:18:48.173 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T14:18:48.173 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T14:18:48.173 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T14:18:48.173 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T14:18:48.173 INFO:tasks.ceph:[global] ms type = async 2024-03-25T14:18:48.173 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T14:18:48.173 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T14:18:48.173 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T14:18:48.173 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T14:18:48.173 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T14:18:48.173 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T14:18:48.174 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T14:18:48.174 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T14:18:48.174 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T14:18:48.174 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T14:18:48.174 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T14:18:48.174 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T14:18:48.174 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T14:18:48.175 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T14:18:48.176 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T14:18:48.176 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T14:18:48.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T14:18:48.233 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T14:18:48.236 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T14:18:48.281 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T14:18:48.308 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.63:6789]'), ('mon.c', '[v1:172.21.15.63:6790]'), ('mon.b', '[v1:172.21.15.153:6789]')] 2024-03-25T14:18:48.309 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.63:6789],[v1:172.21.15.63:6790],[v1:172.21.15.153:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T14:18:48.310 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:48.310 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T14:18:48.364 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.63:6789]' --addv c '[v1:172.21.15.63:6790]' --addv b '[v1:172.21.15.153:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:18:48.450 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:18:48.450 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: generated fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:18:48.450 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = quincy 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 0 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:min_mon_release 17 (quincy) 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:election_strategy: 3 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:18:48.451 INFO:teuthology.orchestra.run.smithi063.stdout:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:18:48.452 INFO:teuthology.orchestra.run.smithi063.stdout:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:18:48.452 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T14:18:48.453 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T14:18:48.468 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d1a7f567-bb16-4bed-8dc7-9f0ba166adec... 2024-03-25T14:18:48.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T14:18:48.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T14:18:48.559 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-03-25T14:18:48.559 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-03-25T14:18:48.559 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T14:18:48.559 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2024-03-25T14:18:48.559 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.560 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T14:18:48.561 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = "[v1:172.21.15.63:6789],[v1:172.21.15.63:6790],[v1:172.21.15.153:6789]" 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: mon client directed command retry = 5 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: mon election default strategy = 3 2024-03-25T14:18:48.562 INFO:teuthology.orchestra.run.smithi063.stdout: ms bind msgr2 = False 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject socket failures = 5000 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: ms type = async 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: osd heartbeat grace = 80 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-03-25T14:18:48.563 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug trim objects = true 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-03-25T14:18:48.564 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: bdev async discard = True 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: bdev enable discard = True 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore allocator = stupid 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore block size = 96636764160 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore fsck on mount = True 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore zero block detection = True 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluefs = 1 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluestore = 1 2024-03-25T14:18:48.565 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 20 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: debug rocksdb = 10 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd full ratio = 0.9 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd nearfull ratio = 0.8 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd blocked scrub grace period = 3600 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd class default list = * 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd class load list = * 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd client watch timeout = 120 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify cached snaps = True 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify missing on start = True 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd failsafe full ratio = 0.95 2024-03-25T14:18:48.566 INFO:teuthology.orchestra.run.smithi063.stdout: osd fast shutdown = False 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: osd max markdown count = 1000 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock override recovery settings = True 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: osd objectstore = bluestore 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue = debug_random 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue cut off = debug_random 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: osd shutdown pgref assert = True 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-03-25T14:18:48.567 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:18:48.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-03-25T14:18:48.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon min osdmap epochs = 50 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd crush smoke test = False 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune interval = 2 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune min = 15 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune txsize = 2 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub interval = 300 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on pool no app = False 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: paxos service trim min = 10 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-03-25T14:18:48.570 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 20 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug objecter = 20 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug rados = 20 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.c] 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.b] 2024-03-25T14:18:48.571 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-03-25T14:18:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-03-25T14:18:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-03-25T14:18:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "[v1:172.21.15.63:6789],[v1:172.21.15.63:6790],[v1:172.21.15.153:6789]" 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 3 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr2 = False 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 5000 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: osd heartbeat grace = 80 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-03-25T14:18:48.576 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: bdev async discard = True 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: bdev enable discard = True 2024-03-25T14:18:48.577 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore allocator = stupid 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 1 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 1 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: osd blocked scrub grace period = 3600 2024-03-25T14:18:48.578 INFO:teuthology.orchestra.run.smithi153.stdout: osd class default list = * 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd class load list = * 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd client watch timeout = 120 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd fast shutdown = False 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd max markdown count = 1000 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: osd shutdown pgref assert = True 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-03-25T14:18:48.579 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-03-25T14:18:48.580 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-03-25T14:18:48.581 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 50 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd crush smoke test = False 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on pool no app = False 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 10 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-03-25T14:18:48.582 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout: debug objecter = 20 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout: debug rados = 20 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-03-25T14:18:48.583 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-03-25T14:18:48.585 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T14:18:48.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T14:18:48.681 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T14:18:48.681 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:48.681 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T14:18:48.695 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:48.696 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T14:18:48.751 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:18:48.751 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:48.751 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T14:18:48.751 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T14:18:48.829 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:18:48.829 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:18:48.884 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:18:48.885 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:18:48.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T14:18:48.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T14:18:48.924 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:18:48.924 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:18:48.979 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T14:18:48.979 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T14:18:48.979 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T14:18:49.033 INFO:teuthology.orchestra.run.smithi063.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T14:18:49.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T14:18:49.095 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T14:18:49.097 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T14:18:49.098 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T14:18:49.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T14:18:49.139 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T14:18:49.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T14:18:49.197 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T14:18:49.212 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T14:18:49.212 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T14:18:49.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T14:18:49.238 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:18:49.239 INFO:tasks.ceph:role: osd.0 2024-03-25T14:18:49.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:18:49.239 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:18:49.313 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:18:49.822 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T14:18:49.824 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T14:18:49.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T14:18:49.977 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T14:18:50.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T14:18:50.077 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:18:50.078 INFO:tasks.ceph:role: osd.1 2024-03-25T14:18:50.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:18:50.078 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:18:50.152 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:18:50.153 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:18:50.153 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:18:50.594 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T14:18:50.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T14:18:50.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T14:18:50.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T14:18:50.722 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T14:18:50.791 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:18:50.792 INFO:tasks.ceph:role: osd.2 2024-03-25T14:18:50.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:18:50.792 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T14:18:50.865 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:18:50.865 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:18:50.865 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:18:50.865 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:18:50.865 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:18:50.865 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:18:50.866 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:18:50.866 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:18:50.866 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:18:50.866 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:18:51.349 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T14:18:51.351 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T14:18:51.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T14:18:51.490 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T14:18:51.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T14:18:51.589 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T14:18:51.590 INFO:tasks.ceph:role: osd.3 2024-03-25T14:18:51.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:18:51.590 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:18:51.664 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:18:51.665 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:18:51.665 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:18:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-03-25T14:18:52.188 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-03-25T14:18:52.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T14:18:52.332 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T14:18:52.366 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:52.464+0000 7fcc05ab5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T14:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:52.464+0000 7fcc05ab5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T14:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:52.464+0000 7fcc05ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T14:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:52.464+0000 7fcc05ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T14:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:52.464+0000 7fcc05ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T14:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:52.465+0000 7fcc05ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T14:18:55.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T14:18:55.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:18:55.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:55.443+0000 7fa9f1a48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T14:18:55.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:55.443+0000 7fa9f1a48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T14:18:55.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:55.443+0000 7fa9f1a48640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T14:18:55.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:55.443+0000 7fa9f1a48640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T14:18:55.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:55.443+0000 7fa9f1a48640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T14:18:55.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:55.444+0000 7fa9f1a48640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T14:18:58.293 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T14:18:58.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:18:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:58.414+0000 7f3d2661e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T14:18:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:58.414+0000 7f3d2661e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T14:18:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:58.414+0000 7f3d2661e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T14:18:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:58.414+0000 7f3d2661e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T14:18:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:58.414+0000 7f3d2661e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T14:18:58.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:18:58.415+0000 7f3d2661e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T14:19:01.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T14:19:01.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:19:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:01.388+0000 7ff03ea68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T14:19:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:01.389+0000 7ff03ea68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T14:19:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:01.389+0000 7ff03ea68640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T14:19:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:01.389+0000 7ff03ea68640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T14:19:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:01.389+0000 7ff03ea68640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T14:19:01.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:01.389+0000 7ff03ea68640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T14:19:04.250 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T14:19:04.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T14:19:04.314 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:19:04.314 INFO:tasks.ceph:role: osd.4 2024-03-25T14:19:04.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:19:04.315 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T14:19:04.392 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:19:04.393 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:19:04.914 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T14:19:04.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T14:19:04.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T14:19:05.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T14:19:05.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T14:19:05.173 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:19:05.173 INFO:tasks.ceph:role: osd.5 2024-03-25T14:19:05.173 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:19:05.173 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T14:19:05.250 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:19:05.250 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:19:05.251 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:19:05.736 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T14:19:05.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T14:19:05.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T14:19:05.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T14:19:05.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T14:19:05.931 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:19:05.931 INFO:tasks.ceph:role: osd.6 2024-03-25T14:19:05.931 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:19:05.932 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T14:19:06.007 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:19:06.007 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:19:06.007 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:19:06.007 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:19:06.007 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:19:06.008 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:19:06.008 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:19:06.008 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:19:06.008 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:19:06.008 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:19:06.526 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T14:19:06.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T14:19:06.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T14:19:06.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T14:19:06.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T14:19:06.758 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T14:19:06.759 INFO:tasks.ceph:role: osd.7 2024-03-25T14:19:06.759 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:19:06.759 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T14:19:06.833 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T14:19:06.834 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T14:19:07.353 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T14:19:07.355 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T14:19:07.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T14:19:07.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T14:19:07.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:19:07.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:07.584+0000 7f4091e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T14:19:07.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:07.584+0000 7f4091e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T14:19:07.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:07.584+0000 7f4091e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T14:19:07.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:07.584+0000 7f4091e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T14:19:07.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:07.584+0000 7f4091e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T14:19:07.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:07.585+0000 7f4091e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T14:19:10.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T14:19:10.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:19:10.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:10.547+0000 7ff5fc9f8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T14:19:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:10.548+0000 7ff5fc9f8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T14:19:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:10.548+0000 7ff5fc9f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T14:19:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:10.548+0000 7ff5fc9f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T14:19:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:10.548+0000 7ff5fc9f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T14:19:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:10.548+0000 7ff5fc9f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T14:19:13.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T14:19:13.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:19:13.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:13.520+0000 7f0202213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T14:19:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:13.521+0000 7f0202213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T14:19:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:13.521+0000 7f0202213640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T14:19:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:13.521+0000 7f0202213640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T14:19:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:13.521+0000 7f0202213640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T14:19:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:13.521+0000 7f0202213640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T14:19:16.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T14:19:16.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:19:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:16.486+0000 7fb20b80d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T14:19:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:16.486+0000 7fb20b80d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T14:19:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:16.486+0000 7fb20b80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T14:19:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:16.486+0000 7fb20b80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T14:19:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:16.486+0000 7fb20b80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T14:19:16.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:16.487+0000 7fb20b80d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T14:19:19.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T14:19:19.357 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T14:19:19.357 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:19.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T14:19:19.389 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:19.390 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T14:19:19.460 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:19.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T14:19:19.529 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:19.529 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T14:19:19.598 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:19.598 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T14:19:19.666 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:19.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T14:19:19.695 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:19.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T14:19:19.763 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:19.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T14:19:19.831 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:19.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T14:19:19.899 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:19.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T14:19:19.966 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:19.966 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T14:19:19.983 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:19.983 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T14:19:20.021 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T14:19:20.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T14:19:20.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T14:19:20.065 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.y] 2024-03-25T14:19:20.065 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDJhwFm7s/1ARAAstjqYRYSWoeOpj7dAGmOYA== 2024-03-25T14:19:20.065 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-03-25T14:19:20.066 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDMhwFmC6a5GxAAhFQ/1PaHyT1tcxqezOaPZg== 2024-03-25T14:19:20.066 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-03-25T14:19:20.066 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDPhwFmMWiBGhAANTjXEP8aJxUB7YhW2+XEMQ== 2024-03-25T14:19:20.066 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-03-25T14:19:20.066 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDShwFmuRvFGBAAWeZbiiQQ02LHtJdG8shz/g== 2024-03-25T14:19:20.066 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.3] 2024-03-25T14:19:20.066 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDVhwFm5Z8+FxAAr4p0GngosRZnGvuK96xTIg== 2024-03-25T14:19:20.067 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2024-03-25T14:19:20.067 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDJhwFm9AGgBRAAzoLoolKoXRQYaEa/OFfo3g== 2024-03-25T14:19:20.067 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.4] 2024-03-25T14:19:20.067 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDbhwFm8GvfIhAAF9OykKyQbtbhKvUgmHl6aw== 2024-03-25T14:19:20.067 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.5] 2024-03-25T14:19:20.067 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDehwFmJvavIBAA+j9upOZRFIYw2gtZbVzHPQ== 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.6] 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDhhwFmpMkQHxAA0b5VXrclqPsJYuKIkWFxLQ== 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.7] 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDkhwFmnVQDHRAAAC8H8JvPpZoLimJe8Am1Cw== 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDJhwFmF8dHCBAA5BTLopGM6Vgbwq1hZR6/mg== 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout:[client.1] 2024-03-25T14:19:20.068 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDJhwFmdJu1CxAAuvCL/EWZvR16pXBZyYRsUg== 2024-03-25T14:19:20.086 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.y] 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDJhwFm7s/1ARAAstjqYRYSWoeOpj7dAGmOYA== 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDMhwFmC6a5GxAAhFQ/1PaHyT1tcxqezOaPZg== 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDPhwFmMWiBGhAANTjXEP8aJxUB7YhW2+XEMQ== 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDShwFmuRvFGBAAWeZbiiQQ02LHtJdG8shz/g== 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDVhwFm5Z8+FxAAr4p0GngosRZnGvuK96xTIg== 2024-03-25T14:19:20.087 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDJhwFm9AGgBRAAzoLoolKoXRQYaEa/OFfo3g== 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDbhwFm8GvfIhAAF9OykKyQbtbhKvUgmHl6aw== 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDehwFmJvavIBAA+j9upOZRFIYw2gtZbVzHPQ== 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.6] 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDhhwFmpMkQHxAA0b5VXrclqPsJYuKIkWFxLQ== 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.7] 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDkhwFmnVQDHRAAAC8H8JvPpZoLimJe8Am1Cw== 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDJhwFmF8dHCBAA5BTLopGM6Vgbwq1hZR6/mg== 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-03-25T14:19:20.088 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDJhwFmdJu1CxAAuvCL/EWZvR16pXBZyYRsUg== 2024-03-25T14:19:20.089 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:19:20.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:19:20.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.238 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.293 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.442 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:19:20.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T14:19:20.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.606 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T14:19:20.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:19:20.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:19:20.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:19:20.774 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T14:19:20.828 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T14:19:20.829 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T14:19:20.855 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T14:19:20.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T14:19:20.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T14:19:21.044 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T14:19:21.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T14:19:21.167 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T14:19:21.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T14:19:21.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T14:19:21.320 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:19:21.323 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T14:19:21.375 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T14:19:21.375 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T14:19:21.376 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T14:19:21.376 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T14:19:21.378 INFO:tasks.ceph.mon.a:Started 2024-03-25T14:19:21.378 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T14:19:21.378 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T14:19:21.378 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T14:19:21.380 INFO:tasks.ceph.mon.c:Started 2024-03-25T14:19:21.380 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T14:19:21.380 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T14:19:21.380 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T14:19:21.418 INFO:tasks.ceph.mon.b:Started 2024-03-25T14:19:21.418 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T14:19:21.418 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T14:19:21.419 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T14:19:21.421 INFO:tasks.ceph.mgr.y:Started 2024-03-25T14:19:21.421 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T14:19:21.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T14:19:21.423 INFO:tasks.ceph.mgr.x:Started 2024-03-25T14:19:21.423 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T14:19:21.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T14:19:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.523+0000 7f0dc96c8640 10 monclient: get_monmap_and_config 2024-03-25T14:19:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.523+0000 7f0dc96c8640 10 monclient: build_initial_monmap 2024-03-25T14:19:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.523+0000 7f0dc96c8640 10 monclient: monmap: 2024-03-25T14:19:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T14:19:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:19:21.524339+0000 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:19:21.524339+0000 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:21.526 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.523+0000 7f0dc96c8640 1 Processor -- start 2024-03-25T14:19:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.523+0000 7f0dc96c8640 1 -- start start 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.523+0000 7f0dc96c8640 10 monclient: init 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc40b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 10 monclient(hunting): picked mon.noname-c con 0x7f0dc4146450 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc406b100 con 0x7f0dc40b45a0 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40b2940 con 0x7f0dc4146450 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc413fec0 con 0x7f0dc40a9920 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc96c8640 10 monclient(hunting): authenticate will time out at 1324.669008s 2024-03-25T14:19:21.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0dc413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:21.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0dc4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:21.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.524+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0dc414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:21.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.725+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0dc4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:21.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.725+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0dc413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:21.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:21.725+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0dc414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:22.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:22.126+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0dc413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:22.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:22.126+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0dc4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:22.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:22.126+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0dc414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:22.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:22.927+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0dc4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:22.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:22.927+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0dc413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:22.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:22.927+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0dc414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:24.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0dc413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0dc414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0dc4145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc40b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0dc4146450 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:24.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc406b100 con 0x7f0dc40b45a0 2024-03-25T14:19:24.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40b2940 con 0x7f0dc4146450 2024-03-25T14:19:24.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:24.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc413fec0 con 0x7f0dc40a9920 2024-03-25T14:19:24.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:24.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:24.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.524+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0db000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:24.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.725+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:24.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.725+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:24.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:24.725+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0db000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:25.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:25.126+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:25.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:25.126+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:25.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:25.126+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0db000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:25.927+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:25.927+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:25.927+0000 7f0dc37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0db000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:27.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0dc4146450 legacy=0x7f0db000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db000fdf0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc40b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0010e20 con 0x7f0dc40b45a0 2024-03-25T14:19:27.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc406b100 con 0x7f0db000fdf0 2024-03-25T14:19:27.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40b2940 con 0x7f0dc40a9920 2024-03-25T14:19:27.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:27.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:27.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc37fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:27.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.524+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 legacy=0x7f0db00101d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:27.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.725+0000 7f0dc37fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:27.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.725+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:27.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:27.725+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 legacy=0x7f0db00101d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:28.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:28.126+0000 7f0dc37fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:28.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:28.126+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:28.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:28.126+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 legacy=0x7f0db00101d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:28.403 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:00:06.864 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:19:28.709 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:00:07.170 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:19:28.853 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:00:07.367 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:19:28.854 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:00:07.369 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:19:28.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:28.927+0000 7f0dc27fc640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-03-25T14:19:28.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:28.927+0000 7f0dc37fe640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6790/0 2024-03-25T14:19:28.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:28.927+0000 7f0dc2ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 legacy=0x7f0db00101d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T14:19:29.179 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:00:07.694 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:19:29.181 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:00:07.696 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:19:30.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:30.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:30.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:30.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 legacy=0x7f0db00101d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.524+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db000fdf0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc40b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.525+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc413fec0 con 0x7f0dc40b45a0 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.525+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0010e20 con 0x7f0db000fdf0 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.525+0000 7f0dc0ff9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc406b100 con 0x7f0dc40a9920 2024-03-25T14:19:30.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.525+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:30.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.572+0000 7f0dc2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 0x7f0db00101d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:49750/0 (socket says 172.21.15.63:49750) 2024-03-25T14:19:30.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.572+0000 7f0dc2ffd640 1 -- 172.21.15.63:0/2216997658 learned_addr learned my addr 172.21.15.63:0/2216997658 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:19:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.912+0000 7f0dc27fc640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T14:19:30.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.912+0000 7f0dc27fc640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T14:19:30.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.912+0000 7f0dc27fc640 1 --1- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 0x7f0db00056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T14:19:30.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.957+0000 7f0dc37fe640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T14:19:30.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.957+0000 7f0dc37fe640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:19:30.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.957+0000 7f0dc37fe640 1 --1- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 0x7f0db0008ea0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-25T14:19:30.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.957+0000 7f0dc37fe640 1 --1- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 0x7f0db0008ea0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:19:30.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:30.957+0000 7f0dc1ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6790/0 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 legacy=0x7f0db00101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0dc40b45a0 legacy=0x7f0db00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db000fdf0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0db000d600 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40b2940 con 0x7f0db000d600 2024-03-25T14:19:33.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00056d0 con 0x7f0db000fdf0 2024-03-25T14:19:33.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00058b0 con 0x7f0dc40a9920 2024-03-25T14:19:33.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:33.525+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db0008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:36.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db000fdf0 legacy=0x7f0db00101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0db000d600 legacy=0x7f0db000d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0db000e0f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db00130a0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0005b00 con 0x7f0db000e0f0 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00102c0 con 0x7f0db00130a0 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00104f0 con 0x7f0dc40a9920 2024-03-25T14:19:36.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:36.525+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:39.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:39.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db000d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db00130a0 legacy=0x7f0db00095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0db000e0f0 legacy=0x7f0db000e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db0008ee0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0db0004de0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc40a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0010720 con 0x7f0db0004de0 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0005260 con 0x7f0db0008ee0 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00053f0 con 0x7f0dc40a9920 2024-03-25T14:19:39.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:39.526+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0dc40a9920 legacy=0x7f0db000d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db0008ee0 legacy=0x7f0db00046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0db0004de0 legacy=0x7f0db000cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0db000f670 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0db0011090 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db0008ee0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db000f4e0 con 0x7f0db000f670 2024-03-25T14:19:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db000fac0 con 0x7f0db0008ee0 2024-03-25T14:19:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0012c30 con 0x7f0db0011090 2024-03-25T14:19:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:42.526+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:45.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:45.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:45.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:45.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:45.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0db0011090 legacy=0x7f0db0011470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:45.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db0008ee0 legacy=0x7f0db000d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:45.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0db000f670 legacy=0x7f0db0009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0db00036c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db00041b0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0db0004740 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0012e60 con 0x7f0db00036c0 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0004c10 con 0x7f0db00041b0 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0011510 con 0x7f0db0004740 2024-03-25T14:19:45.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.527+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:45.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.528+0000 7f0dc27fc640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 legacy=0x7f0db000cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T14:19:45.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.528+0000 7f0dc27fc640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 legacy=0x7f0db000cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:19:45.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.528+0000 7f0dc27fc640 1 --1- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 0x7f0db000cce0 :-1 s=OPENED pgs=29 cs=1 l=1).handle_message read tag failed 2024-03-25T14:19:45.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.528+0000 7f0dc27fc640 1 --1- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 0x7f0db000cce0 :-1 s=OPENED pgs=29 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:19:45.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:45.528+0000 7f0dc1ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6789/0 2024-03-25T14:19:48.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient(hunting): tick 2024-03-25T14:19:48.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T14:19:48.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 1 monclient(hunting): continuing hunt 2024-03-25T14:19:48.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T14:19:48.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0db0004740 legacy=0x7f0db000d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 legacy=0x7f0db000cce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0db00036c0 legacy=0x7f0db0003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0db0009510 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f0db00041b0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f0db0011970 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0011740 con 0x7f0db0011970 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0009990 con 0x7f0db00041b0 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0003aa0 con 0x7f0db0009510 2024-03-25T14:19:48.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:48.527+0000 7f0dc0ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:50.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.067+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122964198 0 0) 0x7f0db0011740 con 0x7f0db0011970 2024-03-25T14:19:50.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.067+0000 7f0dc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:50.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.067+0000 7f0dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:50.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.067+0000 7f0dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:50.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.067+0000 7f0dc1ffb640 10 monclient(hunting): my global_id is 4101 2024-03-25T14:19:50.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.067+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dac003520 con 0x7f0db0011970 2024-03-25T14:19:50.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.090+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971781066 0 0) 0x7f0db0009990 con 0x7f0db00041b0 2024-03-25T14:19:50.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.090+0000 7f0dc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:50.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.090+0000 7f0dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:50.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.090+0000 7f0dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:50.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.090+0000 7f0dc1ffb640 10 monclient(hunting): my global_id is 4105 2024-03-25T14:19:50.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.090+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0011740 con 0x7f0db00041b0 2024-03-25T14:19:50.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.091+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385207660 0 0) 0x7f0db0003aa0 con 0x7f0db0009510 2024-03-25T14:19:50.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.091+0000 7f0dc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:50.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.091+0000 7f0dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:50.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.091+0000 7f0dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:50.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.091+0000 7f0dc1ffb640 10 monclient(hunting): my global_id is 4106 2024-03-25T14:19:50.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.091+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0009990 con 0x7f0db0009510 2024-03-25T14:19:50.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903785949 0 0) 0x7f0dac003520 con 0x7f0db0011970 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6790/0 conn(0x7f0db0009510 legacy=0x7f0db000e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 legacy=0x7f0db000cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 1 monclient: found mon.noname-a 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0003c30 con 0x7f0db0011970 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 10 monclient: _finish_auth 0 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:50.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0dbc004380 con 0x7f0db0011970 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc96c8640 5 monclient: authenticate success, global_id 4101 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc1ffb640 10 monclient: dump: 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:50.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.153+0000 7f0dc96c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dbc004550 con 0x7f0db0011970 2024-03-25T14:19:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc1ffb640 1 -- 172.21.15.63:0/2216997658 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0dbc004920 con 0x7f0db0011970 2024-03-25T14:19:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:50.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc1ffb640 10 monclient: dump: 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 4 set_mon_vals no callback set 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 10 monclient: shutdown 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2216997658 >> v1:172.21.15.63:6789/0 conn(0x7f0db0011970 legacy=0x7f0db000d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2216997658 shutdown_connections 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2216997658 >> 172.21.15.63:0/2216997658 conn(0x7f0dc40b7f40 msgr2=0x7f0dc40b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2216997658 shutdown_connections 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.205+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2216997658 wait complete. 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.206+0000 7f0dc96c8640 10 monclient: build_initial_monmap 2024-03-25T14:19:50.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.206+0000 7f0dc96c8640 10 monclient: monmap: 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.207+0000 7f0dc96c8640 1 librados: starting msgr at 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.207+0000 7f0dc96c8640 1 librados: starting objecter 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.207+0000 7f0dc96c8640 1 Processor -- start 2024-03-25T14:19:50.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.207+0000 7f0dc96c8640 1 -- start start 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.207+0000 7f0dc96c8640 1 librados: setting wanted keys 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.207+0000 7f0dc96c8640 1 librados: calling monclient init 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.207+0000 7f0dc96c8640 10 monclient: init 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 10 monclient(hunting): picked mon.noname-b con 0x7f0db00041b0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 10 monclient(hunting): picked mon.noname-c con 0x7f0db0009510 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 10 monclient(hunting): picked mon.noname-a con 0x7f0db0011970 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc4140550 con 0x7f0db0011970 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40ac310 con 0x7f0db00041b0 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc4138b30 con 0x7f0db0009510 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:50.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.208+0000 7f0dc96c8640 10 monclient(hunting): authenticate will time out at 1353.352644s 2024-03-25T14:19:50.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.212+0000 7f0dc2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 0x7f0dc41daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36370/0 (socket says 172.21.15.63:36370) 2024-03-25T14:19:50.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.212+0000 7f0dc2ffd640 1 -- 172.21.15.63:0/2983900257 learned_addr learned my addr 172.21.15.63:0/2983900257 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:19:50.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.213+0000 7f0dab7fe640 10 client.?.objecter ms_handle_connect 0x7f0db00041b0 2024-03-25T14:19:50.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.213+0000 7f0dab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:50.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.229+0000 7f0dab7fe640 10 client.?.objecter ms_handle_connect 0x7f0db0009510 2024-03-25T14:19:50.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.229+0000 7f0dab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:50.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.244+0000 7f0dab7fe640 10 client.?.objecter ms_handle_connect 0x7f0db0011970 2024-03-25T14:19:50.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.244+0000 7f0dab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:50.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.297+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133920934 0 0) 0x7f0db0009990 con 0x7f0db00041b0 2024-03-25T14:19:50.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.297+0000 7f0dab7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:19:50.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.297+0000 7f0dab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:50.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.297+0000 7f0dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:50.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.297+0000 7f0dab7fe640 10 monclient(hunting): my global_id is 4114 2024-03-25T14:19:50.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.297+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0003aa0 con 0x7f0db00041b0 2024-03-25T14:19:50.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.300+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399020474 0 0) 0x7f0db0011740 con 0x7f0db0009510 2024-03-25T14:19:50.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.300+0000 7f0dab7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:19:50.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.300+0000 7f0dab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:50.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.300+0000 7f0dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:50.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.300+0000 7f0dab7fe640 10 monclient(hunting): my global_id is 4115 2024-03-25T14:19:50.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.300+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0009990 con 0x7f0db0009510 2024-03-25T14:19:50.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.319+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3659703395 0 0) 0x7f0db0009990 con 0x7f0db0009510 2024-03-25T14:19:50.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.319+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db0011740 con 0x7f0db0009510 2024-03-25T14:19:50.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.319+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0db80040a0 con 0x7f0db0009510 2024-03-25T14:19:50.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.319+0000 7f0dab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.319+0000 7f0dab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.319+0000 7f0dab7fe640 10 monclient(hunting): dump: 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.319+0000 7f0dab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:50.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.334+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767493056 0 0) 0x7f0db0003c30 con 0x7f0db0011970 2024-03-25T14:19:50.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.334+0000 7f0dab7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:19:50.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.334+0000 7f0dab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:50.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.334+0000 7f0dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:50.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.334+0000 7f0dab7fe640 10 monclient(hunting): my global_id is 4113 2024-03-25T14:19:50.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.334+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0009990 con 0x7f0db0011970 2024-03-25T14:19:50.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.348+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3239336999 0 0) 0x7f0db0009990 con 0x7f0db0011970 2024-03-25T14:19:50.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.348+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db0003c30 con 0x7f0db0011970 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.348+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0dbc004860 con 0x7f0db0011970 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.348+0000 7f0dab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.348+0000 7f0dab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.348+0000 7f0dab7fe640 10 monclient(hunting): dump: 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.351 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.352 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:50.352 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:50.352 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:50.352 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:50.352 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:50.352 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3360802533 0 0) 0x7f0db0011740 con 0x7f0db0009510 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 >> v1:172.21.15.153:6789/0 conn(0x7f0db00041b0 legacy=0x7f0dc41daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 >> v1:172.21.15.63:6789/0 conn(0x7f0db0011970 legacy=0x7f0dc4138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 1 monclient: found mon.c 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:50.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40ad010 con 0x7f0db0009510 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 10 monclient: _finish_auth 0 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 5 monclient: authenticate success, global_id 4115 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 10 monclient: _renew_subs 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc40ac310 con 0x7f0db0009510 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 10 monclient: _renew_subs 2024-03-25T14:19:50.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:50.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc4140550 con 0x7f0db0009510 2024-03-25T14:19:50.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.349+0000 7f0dc96c8640 1 librados: init done 2024-03-25T14:19:50.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.350+0000 7f0dc96c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:50.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.350+0000 7f0dc96c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:50.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.350+0000 7f0dc96c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:50.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.350+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc4140550 con 0x7f0db0009510 2024-03-25T14:19:50.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0db80044b0 con 0x7f0db0009510 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0db8004880 con 0x7f0db0009510 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dab7fe640 10 monclient: dump: 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:50.640 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0db8004e30 con 0x7f0db0009510 2024-03-25T14:19:50.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.637+0000 7f0dc17fa640 4 set_mon_vals no callback set 2024-03-25T14:19:50.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1711572875 0 0) 0x7f0db8006050 con 0x7f0db0009510 2024-03-25T14:19:50.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 10 client.4115.objecter ms_dispatch 0x7f0dc4000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T14:19:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 3 client.4115.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T14:19:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 3 client.4115.objecter handle_osd_map decoding full epoch 1 2024-03-25T14:19:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 20 client.4115.objecter dump_active .. 0 homeless 2024-03-25T14:19:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0db8005360 con 0x7f0db0009510 2024-03-25T14:19:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.649+0000 7f0dab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:50.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.716+0000 7f0dc96c8640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T14:19:50.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.716+0000 7f0dc96c8640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T14:19:50.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.716+0000 7f0dc96c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:50.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:50.716+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0dc4140520 con 0x7f0db0009510 2024-03-25T14:19:50.765 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:50.763+0000 7ff6d0038040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T14:19:50.823 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:50.820+0000 7ff6d0038040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T14:19:50.866 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:50.865+0000 7f3e02017040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T14:19:50.920 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:50.919+0000 7f3e02017040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T14:19:51.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.091+0000 7f0dab7fe640 1 -- 172.21.15.63:0/2983900257 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f0db80055a0 con 0x7f0db0009510 2024-03-25T14:19:51.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.091+0000 7f0dab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T14:19:51.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.091+0000 7f0dab7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T14:19:51.094 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-03-25T14:19:51.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 10 librados: watch_flush enter 2024-03-25T14:19:51.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 10 librados: watch_flush exit 2024-03-25T14:19:51.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 20 client.4115.objecter shutdown clearing up homeless session... 2024-03-25T14:19:51.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 10 client.4115.objecter successfully canceled tick 2024-03-25T14:19:51.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 10 monclient: shutdown 2024-03-25T14:19:51.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:51.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 >> v1:172.21.15.63:6790/0 conn(0x7f0db0009510 legacy=0x7f0dc4137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:51.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 shutdown_connections 2024-03-25T14:19:51.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 >> 172.21.15.63:0/2983900257 conn(0x7f0dc40b7f40 msgr2=0x7f0dc413f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:51.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 shutdown_connections 2024-03-25T14:19:51.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 1 -- 172.21.15.63:0/2983900257 wait complete. 2024-03-25T14:19:51.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:51.092+0000 7f0dc96c8640 1 librados: shutdown 2024-03-25T14:19:51.106 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T14:19:51.106 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T14:19:51.107 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:51.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T14:19:51.135 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:51.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T14:19:51.206 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:51.207 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T14:19:51.274 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-25T14:19:51.274 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T14:19:51.342 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:51.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T14:19:51.370 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:51.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T14:19:51.417 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.415+0000 7ff6d0038040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T14:19:51.437 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:51.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T14:19:51.458 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.457+0000 7f3e02017040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T14:19:51.476 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.473+0000 7ff6d0038040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T14:19:51.485 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T14:19:51.485 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T14:19:51.518 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.516+0000 7f3e02017040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T14:19:51.523 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.521+0000 7ff6d0038040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T14:19:51.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 40c78739-8253-4432-bca1-f84c1a0e50a6 0 2024-03-25T14:19:51.566 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.564+0000 7f3e02017040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T14:19:51.581 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.579+0000 7ff6d0038040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T14:19:51.623 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.621+0000 7f3e02017040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T14:19:51.625 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.623+0000 7ff6d0038040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T14:19:51.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.664+0000 7ff68370e640 10 monclient: get_monmap_and_config 2024-03-25T14:19:51.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.664+0000 7ff68370e640 10 monclient: build_initial_monmap 2024-03-25T14:19:51.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.664+0000 7ff68370e640 10 monclient: monmap: 2024-03-25T14:19:51.666 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:51.666 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:51.666 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:51.664998+0000 2024-03-25T14:19:51.667 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:51.664998+0000 2024-03-25T14:19:51.667 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:51.667 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:51.667 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:51.667 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:51.667 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:51.667 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 1 Processor -- start 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 1 -- start start 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient: init 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c0a9920 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c0b45a0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c146450 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c06b100 con 0x7ff67c0a9920 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0b2940 con 0x7ff67c0b45a0 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c13fec0 con 0x7ff67c146450 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.665+0000 7ff68370e640 10 monclient(hunting): authenticate will time out at 1348.815536s 2024-03-25T14:19:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.666+0000 7ff6814aa640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff67c0a9920 0x7ff67c13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60628/0 (socket says 172.21.15.153:60628) 2024-03-25T14:19:51.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.666+0000 7ff6814aa640 1 -- 172.21.15.153:0/2753913394 learned_addr learned my addr 172.21.15.153:0/2753913394 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:51.670 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.666+0000 7f3e02017040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.668+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960856045 0 0) 0x7ff67c0b2940 con 0x7ff67c0b45a0 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.668+0000 7ff673fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.668+0000 7ff673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): my global_id is 4117 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff668003520 con 0x7ff67c0b45a0 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809057961 0 0) 0x7ff67c06b100 con 0x7ff67c0a9920 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): my global_id is 4116 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c0b2940 con 0x7ff67c0a9920 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089969982 0 0) 0x7ff67c13fec0 con 0x7ff67c146450 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 10 monclient(hunting): my global_id is 4118 2024-03-25T14:19:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.669+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c06b100 con 0x7ff67c146450 2024-03-25T14:19:51.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183760204 0 0) 0x7ff67c0b2940 con 0x7ff67c0a9920 2024-03-25T14:19:51.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 >> v1:172.21.15.63:6790/0 conn(0x7ff67c146450 legacy=0x7ff67c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 >> v1:172.21.15.153:6789/0 conn(0x7ff67c0b45a0 legacy=0x7ff67c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 10 monclient: _finish_hunting 0 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 1 monclient: found mon.noname-a 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c140050 con 0x7ff67c0a9920 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 10 monclient: _finish_auth 0 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 10 monclient: _check_auth_tickets 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff678001670 con 0x7ff67c0a9920 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff68370e640 5 monclient: authenticate success, global_id 4116 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff68370e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:51.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff673fff640 10 monclient: dump: 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:51.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.672+0000 7ff68370e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6780017d0 con 0x7ff67c0a9920 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff673fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff673fff640 1 -- 172.21.15.153:0/2753913394 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff678001a40 con 0x7ff67c0a9920 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff673fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff673fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff673fff640 10 monclient: dump: 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:51.681 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff68370e640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.674+0000 7ff68370e640 4 set_mon_vals no callback set 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.675+0000 7ff68370e640 10 monclient: shutdown 2024-03-25T14:19:51.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.675+0000 7ff68370e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.675+0000 7ff68370e640 1 -- 172.21.15.153:0/2753913394 >> v1:172.21.15.63:6789/0 conn(0x7ff67c0a9920 legacy=0x7ff67c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.675+0000 7ff68370e640 1 -- 172.21.15.153:0/2753913394 shutdown_connections 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.675+0000 7ff68370e640 1 -- 172.21.15.153:0/2753913394 >> 172.21.15.153:0/2753913394 conn(0x7ff67c0b7f40 msgr2=0x7ff67c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.675+0000 7ff68370e640 1 -- 172.21.15.153:0/2753913394 shutdown_connections 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.675+0000 7ff68370e640 1 -- 172.21.15.153:0/2753913394 wait complete. 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 10 monclient: build_initial_monmap 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 10 monclient: monmap: 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:51.683 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 1 librados: starting msgr at 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 1 librados: starting objecter 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 1 Processor -- start 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 1 -- start start 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 1 librados: setting wanted keys 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 1 librados: calling monclient init 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.676+0000 7ff68370e640 10 monclient: init 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c0a9920 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:51.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c0b45a0 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c146450 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c1d74f0 con 0x7ff67c0a9920 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0ac020 con 0x7ff67c146450 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0ac300 con 0x7ff67c0b45a0 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.677+0000 7ff68370e640 10 monclient(hunting): authenticate will time out at 1348.827488s 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.678+0000 7ff6814aa640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff67c0a9920 0x7ff67c1447b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60630/0 (socket says 172.21.15.153:60630) 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.678+0000 7ff6814aa640 1 -- 172.21.15.153:0/126548945 learned_addr learned my addr 172.21.15.153:0/126548945 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.678+0000 7ff671ffb640 10 client.?.objecter ms_handle_connect 0x7ff67c0a9920 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.678+0000 7ff671ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.679+0000 7ff671ffb640 10 client.?.objecter ms_handle_connect 0x7ff67c146450 2024-03-25T14:19:51.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.679+0000 7ff671ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:51.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.680+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225803461 0 0) 0x7ff67c1d74f0 con 0x7ff67c0a9920 2024-03-25T14:19:51.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.680+0000 7ff671ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:51.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.680+0000 7ff671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:51.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.680+0000 7ff671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:51.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.680+0000 7ff671ffb640 10 monclient(hunting): my global_id is 4119 2024-03-25T14:19:51.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.680+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff660004eb0 con 0x7ff67c0a9920 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.681+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414443606 0 0) 0x7ff67c0ac020 con 0x7ff67c146450 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.681+0000 7ff671ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.681+0000 7ff671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.681+0000 7ff671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.681+0000 7ff671ffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.681+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c1d74f0 con 0x7ff67c146450 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.682+0000 7ff671ffb640 10 client.?.objecter ms_handle_connect 0x7ff67c0b45a0 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.682+0000 7ff671ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.684+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 419087153 0 0) 0x7ff660004eb0 con 0x7ff67c0a9920 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.684+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67c0ac020 con 0x7ff67c0a9920 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.684+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff678001ae0 con 0x7ff67c0a9920 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.684+0000 7ff671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.684+0000 7ff671ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:51.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.684+0000 7ff671ffb640 10 monclient(hunting): dump: 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.684+0000 7ff671ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:51.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.685+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697350307 0 0) 0x7ff67c0ac300 con 0x7ff67c0b45a0 2024-03-25T14:19:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.685+0000 7ff671ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.685+0000 7ff671ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.685+0000 7ff671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.685+0000 7ff671ffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T14:19:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.685+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff660004eb0 con 0x7ff67c0b45a0 2024-03-25T14:19:51.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.686+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720377741 0 0) 0x7ff67c1d74f0 con 0x7ff67c146450 2024-03-25T14:19:51.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.686+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67c0ac300 con 0x7ff67c146450 2024-03-25T14:19:51.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.686+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff674001bd0 con 0x7ff67c146450 2024-03-25T14:19:51.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.686+0000 7ff671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:51.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.686+0000 7ff671ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:51.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.686+0000 7ff671ffb640 10 monclient(hunting): dump: 2024-03-25T14:19:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:51.695 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:51.696 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:51.696 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:51.696 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:51.696 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:51.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3626280857 0 0) 0x7ff67c0ac020 con 0x7ff67c0a9920 2024-03-25T14:19:51.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 >> v1:172.21.15.63:6790/0 conn(0x7ff67c0b45a0 legacy=0x7ff67c1d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:51.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 >> v1:172.21.15.153:6789/0 conn(0x7ff67c146450 legacy=0x7ff67c137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:51.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:19:51.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 1 monclient: found mon.a 2024-03-25T14:19:51.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:51.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:51.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c141340 con 0x7ff67c0a9920 2024-03-25T14:19:51.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 10 monclient: _finish_auth 0 2024-03-25T14:19:51.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:19:51.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:51.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 5 monclient: authenticate success, global_id 4119 2024-03-25T14:19:51.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 10 monclient: _renew_subs 2024-03-25T14:19:51.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:51.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67c1d8090 con 0x7ff67c0a9920 2024-03-25T14:19:51.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:51.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 10 monclient: _renew_subs 2024-03-25T14:19:51.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:51.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67c143c00 con 0x7ff67c0a9920 2024-03-25T14:19:51.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.689+0000 7ff68370e640 1 librados: init done 2024-03-25T14:19:51.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.690+0000 7ff68370e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:51.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.690+0000 7ff68370e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:51.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.690+0000 7ff68370e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:51.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.690+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67c141530 con 0x7ff67c0a9920 2024-03-25T14:19:51.754 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.751+0000 7ff6d0038040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T14:19:51.787 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.786+0000 7f3e02017040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T14:19:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff678004a90 con 0x7ff67c0a9920 2024-03-25T14:19:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff671ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:51.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff678004ee0 con 0x7ff67c0a9920 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff671ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff671ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff671ffb640 10 monclient: dump: 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:51.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff6737fe640 4 set_mon_vals no callback set 2024-03-25T14:19:51.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.813+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff678005490 con 0x7ff67c0a9920 2024-03-25T14:19:51.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 127975453 0 0) 0x7ff6780065f0 con 0x7ff67c0a9920 2024-03-25T14:19:51.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 10 client.4119.objecter ms_dispatch 0x7ff67c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T14:19:51.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T14:19:51.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T14:19:51.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T14:19:51.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff678006b60 con 0x7ff67c0a9920 2024-03-25T14:19:51.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:51.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.815+0000 7ff671ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:51.829 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.827+0000 7ff6d0038040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T14:19:51.859 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.857+0000 7f3e02017040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T14:19:51.877 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.875+0000 7ff6d0038040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T14:19:51.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.889+0000 7ff68370e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "40c78739-8253-4432-bca1-f84c1a0e50a6", "id": 0}] 2024-03-25T14:19:51.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.889+0000 7ff68370e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "40c78739-8253-4432-bca1-f84c1a0e50a6", "id": 0}] 2024-03-25T14:19:51.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.889+0000 7ff68370e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:51.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:51.889+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "40c78739-8253-4432-bca1-f84c1a0e50a6", "id": 0} v 0) v1 -- 0x7ff67c04f730 con 0x7ff67c0a9920 2024-03-25T14:19:51.906 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.905+0000 7f3e02017040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T14:19:51.958 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:51.956+0000 7f3e02017040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T14:19:51.961 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:51.959+0000 7ff6d0038040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T14:19:52.005 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:52.003+0000 7f3e02017040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T14:19:52.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.018+0000 7ff671ffb640 1 -- 172.21.15.153:0/126548945 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "40c78739-8253-4432-bca1-f84c1a0e50a6", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 4046699760 0 732707515) 0x7ff678005a40 con 0x7ff67c0a9920 2024-03-25T14:19:52.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.018+0000 7ff671ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "40c78739-8253-4432-bca1-f84c1a0e50a6", "id": 0}] 2024-03-25T14:19:52.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.018+0000 7ff671ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:52.020 INFO:teuthology.orchestra.run.smithi153.stdout:0 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 10 librados: watch_flush enter 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 10 librados: watch_flush exit 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 10 client.4119.objecter successfully canceled tick 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 10 monclient: shutdown 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 >> v1:172.21.15.63:6789/0 conn(0x7ff67c0a9920 legacy=0x7ff67c1447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 shutdown_connections 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 >> 172.21.15.153:0/126548945 conn(0x7ff67c0b7f40 msgr2=0x7ff67c138520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 shutdown_connections 2024-03-25T14:19:52.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 1 -- 172.21.15.153:0/126548945 wait complete. 2024-03-25T14:19:52.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.019+0000 7ff68370e640 1 librados: shutdown 2024-03-25T14:19:52.025 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:52.023+0000 7ff6d0038040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T14:19:52.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 06668c03-387c-4b0b-8489-9e3661b8144f 1 2024-03-25T14:19:52.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.132+0000 7fcab0172640 10 monclient: get_monmap_and_config 2024-03-25T14:19:52.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.132+0000 7fcab0172640 10 monclient: build_initial_monmap 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.132+0000 7fcab0172640 10 monclient: monmap: 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:52.132548+0000 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:52.132548+0000 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:52.135 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:52.136 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:52.136 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.132+0000 7fcab0172640 1 Processor -- start 2024-03-25T14:19:52.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.132+0000 7fcab0172640 1 -- start start 2024-03-25T14:19:52.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.132+0000 7fcab0172640 10 monclient: init 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa80a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa80b45a0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa8146450 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa806b100 con 0x7fcaa8146450 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa80b2940 con 0x7fcaa80b45a0 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa813fec0 con 0x7fcaa80a9920 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.133+0000 7fcab0172640 10 monclient(hunting): authenticate will time out at 1349.283105s 2024-03-25T14:19:52.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.138+0000 7fcaad70d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcaa80b45a0 0x7fcaa8145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36820/0 (socket says 172.21.15.153:36820) 2024-03-25T14:19:52.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.138+0000 7fcaad70d640 1 -- 172.21.15.153:0/2222366134 learned_addr learned my addr 172.21.15.153:0/2222366134 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:52.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.155+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769634135 0 0) 0x7fcaa806b100 con 0x7fcaa8146450 2024-03-25T14:19:52.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.155+0000 7fcaacf0c640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.155+0000 7fcaacf0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.155+0000 7fcaacf0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.155+0000 7fcaacf0c640 10 monclient(hunting): my global_id is 4122 2024-03-25T14:19:52.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.155+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca94003520 con 0x7fcaa8146450 2024-03-25T14:19:52.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.157+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233365859 0 0) 0x7fcaa80b2940 con 0x7fcaa80b45a0 2024-03-25T14:19:52.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.157+0000 7fcaacf0c640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.157+0000 7fcaacf0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.157+0000 7fcaacf0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.157+0000 7fcaacf0c640 10 monclient(hunting): my global_id is 4123 2024-03-25T14:19:52.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.157+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa806b100 con 0x7fcaa80b45a0 2024-03-25T14:19:52.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 678108586 0 0) 0x7fca94003520 con 0x7fcaa8146450 2024-03-25T14:19:52.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 >> v1:172.21.15.63:6790/0 conn(0x7fcaa80a9920 legacy=0x7fcaa813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 >> v1:172.21.15.153:6789/0 conn(0x7fcaa80b45a0 legacy=0x7fcaa8145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 10 monclient: _finish_hunting 0 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 1 monclient: found mon.noname-a 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa8140050 con 0x7fcaa8146450 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 10 monclient: _finish_auth 0 2024-03-25T14:19:52.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 10 monclient: _check_auth_tickets 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcaa0001a10 con 0x7fcaa8146450 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcab0172640 5 monclient: authenticate success, global_id 4122 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcab0172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcaacf0c640 10 monclient: dump: 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.163 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.164 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.159+0000 7fcab0172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:52.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaa0001b70 con 0x7fcaa8146450 2024-03-25T14:19:52.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcaacf0c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:52.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcaacf0c640 1 -- 172.21.15.153:0/2222366134 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcaa0001de0 con 0x7fcaa8146450 2024-03-25T14:19:52.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcaacf0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcaacf0c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:52.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcaacf0c640 10 monclient: dump: 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcab0172640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcab0172640 4 set_mon_vals no callback set 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcab0172640 10 monclient: shutdown 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcab0172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcab0172640 1 -- 172.21.15.153:0/2222366134 >> v1:172.21.15.63:6789/0 conn(0x7fcaa8146450 legacy=0x7fcaa814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.161+0000 7fcab0172640 1 -- 172.21.15.153:0/2222366134 shutdown_connections 2024-03-25T14:19:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.162+0000 7fcab0172640 1 -- 172.21.15.153:0/2222366134 >> 172.21.15.153:0/2222366134 conn(0x7fcaa80b7f40 msgr2=0x7fcaa80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.162+0000 7fcab0172640 1 -- 172.21.15.153:0/2222366134 shutdown_connections 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.162+0000 7fcab0172640 1 -- 172.21.15.153:0/2222366134 wait complete. 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.162+0000 7fcab0172640 10 monclient: build_initial_monmap 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.162+0000 7fcab0172640 10 monclient: monmap: 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:52.175 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 librados: starting msgr at 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 librados: starting objecter 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 Processor -- start 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 -- start start 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 librados: setting wanted keys 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 librados: calling monclient init 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient: init 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient(hunting): picked mon.noname-a con 0x7fcaa80a9920 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient(hunting): picked mon.noname-b con 0x7fcaa80b45a0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient(hunting): picked mon.noname-c con 0x7fcaa8146450 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa8142760 con 0x7fcaa80a9920 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa8135a90 con 0x7fcaa80b45a0 2024-03-25T14:19:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa813cf30 con 0x7fcaa8146450 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.163+0000 7fcab0172640 10 monclient(hunting): authenticate will time out at 1349.313449s 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.164+0000 7fcaad70d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcaa80b45a0 0x7fcaa8134c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36836/0 (socket says 172.21.15.153:36836) 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.164+0000 7fcaad70d640 1 -- 172.21.15.153:0/2588125267 learned_addr learned my addr 172.21.15.153:0/2588125267 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.164+0000 7fca9e7fc640 10 client.?.objecter ms_handle_connect 0x7fcaa80b45a0 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.164+0000 7fca9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.164+0000 7fca9e7fc640 10 client.?.objecter ms_handle_connect 0x7fcaa8146450 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.164+0000 7fca9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931292007 0 0) 0x7fcaa8135a90 con 0x7fcaa80b45a0 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T14:19:52.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca8c004eb0 con 0x7fcaa80b45a0 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 10 client.?.objecter ms_handle_connect 0x7fcaa80a9920 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.166+0000 7fca9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.167+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232056942 0 0) 0x7fcaa813cf30 con 0x7fcaa8146450 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.167+0000 7fca9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.167+0000 7fca9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.167+0000 7fca9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.167+0000 7fca9e7fc640 10 monclient(hunting): my global_id is 4124 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.167+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa8135a90 con 0x7fcaa8146450 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.169+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676274657 0 0) 0x7fcaa8142760 con 0x7fcaa80a9920 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.169+0000 7fca9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.169+0000 7fca9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.169+0000 7fca9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.169+0000 7fca9e7fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T14:19:52.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.169+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa813cf30 con 0x7fcaa80a9920 2024-03-25T14:19:52.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157713118 0 0) 0x7fca8c004eb0 con 0x7fcaa80b45a0 2024-03-25T14:19:52.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa8142760 con 0x7fcaa80b45a0 2024-03-25T14:19:52.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca980019b0 con 0x7fcaa80b45a0 2024-03-25T14:19:52.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:19:52.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 10 monclient(hunting): dump: 2024-03-25T14:19:52.181 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 731781974 0 0) 0x7fcaa8135a90 con 0x7fcaa8146450 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca8c004eb0 con 0x7fcaa8146450 2024-03-25T14:19:52.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcaa0001e80 con 0x7fcaa8146450 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.171+0000 7fca9e7fc640 10 monclient(hunting): dump: 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.183 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3279527847 0 0) 0x7fcaa8142760 con 0x7fcaa80b45a0 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 >> v1:172.21.15.63:6790/0 conn(0x7fcaa8146450 legacy=0x7fcaa8135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 >> v1:172.21.15.63:6789/0 conn(0x7fcaa80a9920 legacy=0x7fcaa8140bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 1 monclient: found mon.b 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa8142f80 con 0x7fcaa80b45a0 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:19:52.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fca9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fcab0172640 5 monclient: authenticate success, global_id 4126 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fcab0172640 10 monclient: _renew_subs 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fcab0172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.174+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa81434a0 con 0x7fcaa80b45a0 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.175+0000 7fcab0172640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.175+0000 7fcab0172640 10 monclient: _renew_subs 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.175+0000 7fcab0172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.175+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa81412e0 con 0x7fcaa80b45a0 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.175+0000 7fcab0172640 1 librados: init done 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.176+0000 7fcab0172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.176+0000 7fcab0172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.176+0000 7fcab0172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.176+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa804f730 con 0x7fcaa80b45a0 2024-03-25T14:19:52.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca980047e0 con 0x7fcaa80b45a0 2024-03-25T14:19:52.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:52.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca98004af0 con 0x7fcaa80b45a0 2024-03-25T14:19:52.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 10 monclient: dump: 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fca98005060 con 0x7fcaa80b45a0 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9ffff640 4 set_mon_vals no callback set 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3890767825 0 0) 0x7fca98006610 con 0x7fcaa80b45a0 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 10 client.4126.objecter ms_dispatch 0x7fcaa8000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fca98006a70 con 0x7fcaa80b45a0 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:52.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.399+0000 7fca9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:52.413 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:52.411+0000 7f3e02017040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T14:19:52.438 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:52.435+0000 7ff6d0038040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T14:19:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.473+0000 7fcab0172640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "06668c03-387c-4b0b-8489-9e3661b8144f", "id": 1}] 2024-03-25T14:19:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.473+0000 7fcab0172640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "06668c03-387c-4b0b-8489-9e3661b8144f", "id": 1}] 2024-03-25T14:19:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.473+0000 7fcab0172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.473+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "06668c03-387c-4b0b-8489-9e3661b8144f", "id": 1} v 0) v1 -- 0x7fcaa8142d60 con 0x7fcaa80b45a0 2024-03-25T14:19:52.552 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:52.550+0000 7f3e02017040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T14:19:52.583 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:52.581+0000 7ff6d0038040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T14:19:52.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.608+0000 7fca9e7fc640 1 -- 172.21.15.153:0/2588125267 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "06668c03-387c-4b0b-8489-9e3661b8144f", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2251144437 0 940483276) 0x7fca98005660 con 0x7fcaa80b45a0 2024-03-25T14:19:52.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.608+0000 7fca9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "06668c03-387c-4b0b-8489-9e3661b8144f", "id": 1}] 2024-03-25T14:19:52.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.608+0000 7fca9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:52.611 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-03-25T14:19:52.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.609+0000 7fcab0172640 10 librados: watch_flush enter 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.609+0000 7fcab0172640 10 librados: watch_flush exit 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.609+0000 7fcab0172640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.609+0000 7fcab0172640 10 client.4126.objecter successfully canceled tick 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.609+0000 7fcab0172640 10 monclient: shutdown 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.609+0000 7fcab0172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.610+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 >> v1:172.21.15.153:6789/0 conn(0x7fcaa80b45a0 legacy=0x7fcaa8134c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.610+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 shutdown_connections 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.610+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 >> 172.21.15.153:0/2588125267 conn(0x7fcaa80b7f40 msgr2=0x7fcaa8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:52.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.610+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 shutdown_connections 2024-03-25T14:19:52.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.610+0000 7fcab0172640 1 -- 172.21.15.153:0/2588125267 wait complete. 2024-03-25T14:19:52.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.610+0000 7fcab0172640 1 librados: shutdown 2024-03-25T14:19:52.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 8962c36e-fad1-4188-8093-f42c888c9df8 2 2024-03-25T14:19:52.696 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:52.695+0000 7f3e02017040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T14:19:52.697 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:52.695+0000 7ff6d0038040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T14:19:52.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.720+0000 7f1535b0e640 10 monclient: get_monmap_and_config 2024-03-25T14:19:52.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.720+0000 7f1535b0e640 10 monclient: build_initial_monmap 2024-03-25T14:19:52.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.720+0000 7f1535b0e640 10 monclient: monmap: 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:52.720524+0000 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:52.720524+0000 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:52.722 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.720+0000 7f1535b0e640 1 Processor -- start 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.720+0000 7f1535b0e640 1 -- start start 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.720+0000 7f1535b0e640 10 monclient: init 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f15300a9920 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f15300b45a0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:52.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f1530146450 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:52.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153006b100 con 0x7f15300b45a0 2024-03-25T14:19:52.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300b2940 con 0x7f15300a9920 2024-03-25T14:19:52.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153013fec0 con 0x7f1530146450 2024-03-25T14:19:52.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:52.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f1535b0e640 10 monclient(hunting): authenticate will time out at 1349.871055s 2024-03-25T14:19:52.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f152effd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f15300b45a0 0x7f1530145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60660/0 (socket says 172.21.15.153:60660) 2024-03-25T14:19:52.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.721+0000 7f152effd640 1 -- 172.21.15.153:0/1979201948 learned_addr learned my addr 172.21.15.153:0/1979201948 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:52.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63088222 0 0) 0x7f153013fec0 con 0x7f1530146450 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): my global_id is 4127 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1518003520 con 0x7f1530146450 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649079079 0 0) 0x7f15300b2940 con 0x7f15300a9920 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): my global_id is 4129 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153013fec0 con 0x7f15300a9920 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015180027 0 0) 0x7f153006b100 con 0x7f15300b45a0 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 10 monclient(hunting): my global_id is 4128 2024-03-25T14:19:52.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.724+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15300b2940 con 0x7f15300b45a0 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568708814 0 0) 0x7f153013fec0 con 0x7f15300a9920 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 >> v1:172.21.15.63:6790/0 conn(0x7f1530146450 legacy=0x7f153014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 >> v1:172.21.15.63:6789/0 conn(0x7f15300b45a0 legacy=0x7f1530145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 1 monclient: found mon.noname-c 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1530140050 con 0x7f15300a9920 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1524001670 con 0x7f15300a9920 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f1535b0e640 5 monclient: authenticate success, global_id 4129 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f1535b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f152e7fc640 10 monclient: dump: 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.731 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.732 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.732 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.732 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.732 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.732 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.732 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.727+0000 7f1535b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:52.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15240017d0 con 0x7f15300a9920 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f152e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f152e7fc640 1 -- 172.21.15.153:0/1979201948 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1524001a40 con 0x7f15300a9920 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f152e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f152e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f152e7fc640 10 monclient: dump: 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.737 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.738 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f1535b0e640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.729+0000 7f1535b0e640 4 set_mon_vals no callback set 2024-03-25T14:19:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 10 monclient: shutdown 2024-03-25T14:19:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 1 -- 172.21.15.153:0/1979201948 >> v1:172.21.15.153:6789/0 conn(0x7f15300a9920 legacy=0x7f153013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 1 -- 172.21.15.153:0/1979201948 shutdown_connections 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 1 -- 172.21.15.153:0/1979201948 >> 172.21.15.153:0/1979201948 conn(0x7f15300b7f40 msgr2=0x7f15300b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 1 -- 172.21.15.153:0/1979201948 shutdown_connections 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 1 -- 172.21.15.153:0/1979201948 wait complete. 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 10 monclient: build_initial_monmap 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.730+0000 7f1535b0e640 10 monclient: monmap: 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:52.739 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.731+0000 7f1535b0e640 1 librados: starting msgr at 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.731+0000 7f1535b0e640 1 librados: starting objecter 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.731+0000 7f1535b0e640 1 Processor -- start 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.731+0000 7f1535b0e640 1 -- start start 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.731+0000 7f1535b0e640 1 librados: setting wanted keys 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.731+0000 7f1535b0e640 1 librados: calling monclient init 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.731+0000 7f1535b0e640 10 monclient: init 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:52.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:19:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f15300a9920 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f15300b45a0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f1530146450 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300ac0e0 con 0x7f15300a9920 2024-03-25T14:19:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300ac330 con 0x7f15300b45a0 2024-03-25T14:19:52.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1530143fe0 con 0x7f1530146450 2024-03-25T14:19:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1535b0e640 10 monclient(hunting): authenticate will time out at 1349.881777s 2024-03-25T14:19:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f152f7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f15300a9920 0x7f1530144500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60668/0 (socket says 172.21.15.153:60668) 2024-03-25T14:19:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f152f7fe640 1 -- 172.21.15.153:0/206946032 learned_addr learned my addr 172.21.15.153:0/206946032 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1517fff640 10 client.?.objecter ms_handle_connect 0x7f15300a9920 2024-03-25T14:19:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1517fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1517fff640 10 client.?.objecter ms_handle_connect 0x7f15300b45a0 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.732+0000 7f1517fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.733+0000 7f1517fff640 10 client.?.objecter ms_handle_connect 0x7f1530146450 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.733+0000 7f1517fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910737662 0 0) 0x7f15300ac0e0 con 0x7f15300a9920 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): my global_id is 4131 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c004eb0 con 0x7f15300a9920 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651886375 0 0) 0x7f15300ac330 con 0x7f15300b45a0 2024-03-25T14:19:52.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 10 monclient(hunting): my global_id is 4132 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.734+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15300ac0e0 con 0x7f15300b45a0 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.735+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621936945 0 0) 0x7f1530143fe0 con 0x7f1530146450 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.735+0000 7f1517fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.735+0000 7f1517fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.735+0000 7f1517fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.735+0000 7f1517fff640 10 monclient(hunting): my global_id is 4130 2024-03-25T14:19:52.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.735+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15300ac330 con 0x7f1530146450 2024-03-25T14:19:52.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 797800663 0 0) 0x7f15300ac0e0 con 0x7f15300b45a0 2024-03-25T14:19:52.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1530143fe0 con 0x7f15300b45a0 2024-03-25T14:19:52.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1520001870 con 0x7f15300b45a0 2024-03-25T14:19:52.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 10 monclient(hunting): dump: 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:52.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226718569 0 0) 0x7f151c004eb0 con 0x7f15300a9920 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15300ac0e0 con 0x7f15300a9920 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1524001ae0 con 0x7f15300a9920 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.739+0000 7f1517fff640 10 monclient(hunting): dump: 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.749 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.740+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63399700 0 0) 0x7f15300ac330 con 0x7f1530146450 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.740+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f151c004eb0 con 0x7f1530146450 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.740+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1528001a30 con 0x7f1530146450 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.740+0000 7f1517fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.740+0000 7f1517fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.740+0000 7f1517fff640 10 monclient(hunting): dump: 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.750 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1372987505 0 0) 0x7f1530143fe0 con 0x7f15300b45a0 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 >> v1:172.21.15.63:6790/0 conn(0x7f1530146450 legacy=0x7f15301e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 >> v1:172.21.15.63:6789/0 conn(0x7f15300a9920 legacy=0x7f1530144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 10 monclient: _finish_hunting 0 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 1 monclient: found mon.b 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1530144c10 con 0x7f15300b45a0 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 10 monclient: _finish_auth 0 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 10 monclient: _check_auth_tickets 2024-03-25T14:19:52.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1517fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1535b0e640 5 monclient: authenticate success, global_id 4132 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1535b0e640 10 monclient: _renew_subs 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1535b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.742+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1530141340 con 0x7f15300b45a0 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 10 monclient: _renew_subs 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15301db990 con 0x7f15300b45a0 2024-03-25T14:19:52.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 1 librados: init done 2024-03-25T14:19:52.752 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f3e02017040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T14:19:52.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:52.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:52.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.743+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1530144170 con 0x7f15300b45a0 2024-03-25T14:19:52.754 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:52.746+0000 7ff6d0038040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1520001c80 con 0x7f15300b45a0 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1520004980 con 0x7f15300b45a0 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 10 monclient: dump: 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:52.847 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1520004f30 con 0x7f15300b45a0 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f152dffb640 4 set_mon_vals no callback set 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 157812067 0 0) 0x7f1520006460 con 0x7f15300b45a0 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 10 client.4132.objecter ms_dispatch 0x7f1530000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T14:19:52.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-25T14:19:52.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T14:19:52.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1520006780 con 0x7f15300b45a0 2024-03-25T14:19:52.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:52.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.844+0000 7f1517fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:52.861 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:52.859+0000 7f3e02017040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:19:52.866 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:52.865+0000 7ff6d0038040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:19:52.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.919+0000 7f1535b0e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8962c36e-fad1-4188-8093-f42c888c9df8", "id": 2}] 2024-03-25T14:19:52.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.919+0000 7f1535b0e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8962c36e-fad1-4188-8093-f42c888c9df8", "id": 2}] 2024-03-25T14:19:52.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.919+0000 7f1535b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:52.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.919+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8962c36e-fad1-4188-8093-f42c888c9df8", "id": 2} v 0) v1 -- 0x7f153004f730 con 0x7f15300b45a0 2024-03-25T14:19:52.959 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:52.957+0000 7f3e02017040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T14:19:52.969 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:52.968+0000 7ff6d0038040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T14:19:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.993+0000 7f1517fff640 1 -- 172.21.15.153:0/206946032 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8962c36e-fad1-4188-8093-f42c888c9df8", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 784773479 0 216599125) 0x7f1520006110 con 0x7f15300b45a0 2024-03-25T14:19:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.993+0000 7f1517fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8962c36e-fad1-4188-8093-f42c888c9df8", "id": 2}] 2024-03-25T14:19:52.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.993+0000 7f1517fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:52.996 INFO:teuthology.orchestra.run.smithi153.stdout:2 2024-03-25T14:19:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 10 librados: watch_flush enter 2024-03-25T14:19:52.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 10 librados: watch_flush exit 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 10 client.4132.objecter successfully canceled tick 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 10 monclient: shutdown 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 >> v1:172.21.15.153:6789/0 conn(0x7f15300b45a0 legacy=0x7f1530137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 shutdown_connections 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.994+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 >> 172.21.15.153:0/206946032 conn(0x7f15300b7f40 msgr2=0x7f1530074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.995+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 shutdown_connections 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.995+0000 7f1535b0e640 1 -- 172.21.15.153:0/206946032 wait complete. 2024-03-25T14:19:52.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:52.995+0000 7f1535b0e640 1 librados: shutdown 2024-03-25T14:19:53.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 07600efd-4d69-477c-a480-b036389aeb1a 3 2024-03-25T14:19:53.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.104+0000 7f8bd1881640 10 monclient: get_monmap_and_config 2024-03-25T14:19:53.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.104+0000 7f8bd1881640 10 monclient: build_initial_monmap 2024-03-25T14:19:53.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.104+0000 7f8bd1881640 10 monclient: monmap: 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:53.105296+0000 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:53.105296+0000 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:53.107 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.105+0000 7f8bd1881640 1 Processor -- start 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.105+0000 7f8bd1881640 1 -- start start 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.105+0000 7f8bd1881640 10 monclient: init 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.105+0000 7f8bd1881640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.105+0000 7f8bd1881640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.105+0000 7f8bd1881640 10 monclient(hunting): picked mon.noname-b con 0x7f8bcc0b4700 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.105+0000 7f8bd1881640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc142a20 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bd1881640 10 monclient(hunting): picked mon.noname-a con 0x7f8bcc0a96b0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:53.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bd1881640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0a9d60 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bd1881640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc13ff10 con 0x7f8bcc142a20 2024-03-25T14:19:53.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bd1881640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0aa1f0 con 0x7f8bcc0b4700 2024-03-25T14:19:53.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bd1881640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:53.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bd1881640 10 monclient(hunting): authenticate will time out at 1350.255685s 2024-03-25T14:19:53.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bcb7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8bcc0a96b0 0x7f8bcc13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60684/0 (socket says 172.21.15.153:60684) 2024-03-25T14:19:53.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bcb7fe640 1 -- 172.21.15.153:0/4179942677 learned_addr learned my addr 172.21.15.153:0/4179942677 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:53.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.106+0000 7f8bca7fc640 1 --1- 172.21.15.153:0/4179942677 >> v1:172.21.15.153:6789/0 conn(0x7f8bcc142a20 0x7f8bcc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36856/0 (socket says 172.21.15.153:36856) 2024-03-25T14:19:53.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460858117 0 0) 0x7f8bcc0a9d60 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): my global_id is 4134 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb4003520 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131963729 0 0) 0x7f8bcc13ff10 con 0x7f8bcc142a20 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): my global_id is 4135 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc0a9d60 con 0x7f8bcc142a20 2024-03-25T14:19:53.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667417473 0 0) 0x7f8bcc0aa1f0 con 0x7f8bcc0b4700 2024-03-25T14:19:53.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 10 monclient(hunting): my global_id is 4133 2024-03-25T14:19:53.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.109+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc13ff10 con 0x7f8bcc0b4700 2024-03-25T14:19:53.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3379470326 0 0) 0x7f8bb4003520 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 >> v1:172.21.15.63:6790/0 conn(0x7f8bcc0b4700 legacy=0x7f8bcc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 >> v1:172.21.15.153:6789/0 conn(0x7f8bcc142a20 legacy=0x7f8bcc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:19:53.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 1 monclient: found mon.noname-a 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc0b4ae0 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 10 monclient: _finish_auth 0 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8bc4001a10 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bd1881640 5 monclient: authenticate success, global_id 4134 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bc9ffb640 10 monclient: dump: 2024-03-25T14:19:53.116 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.117 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.112+0000 7f8bd1881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:53.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc4001b70 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:53.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bc9ffb640 1 -- 172.21.15.153:0/4179942677 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8bc4001f80 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:53.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 4 set_mon_vals no callback set 2024-03-25T14:19:53.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bc9ffb640 10 monclient: dump: 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.123 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 10 monclient: shutdown 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 1 -- 172.21.15.153:0/4179942677 >> v1:172.21.15.63:6789/0 conn(0x7f8bcc0a96b0 legacy=0x7f8bcc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 1 -- 172.21.15.153:0/4179942677 shutdown_connections 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 1 -- 172.21.15.153:0/4179942677 >> 172.21.15.153:0/4179942677 conn(0x7f8bcc0668e0 msgr2=0x7f8bcc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 1 -- 172.21.15.153:0/4179942677 shutdown_connections 2024-03-25T14:19:53.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.114+0000 7f8bd1881640 1 -- 172.21.15.153:0/4179942677 wait complete. 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 10 monclient: build_initial_monmap 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 10 monclient: monmap: 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 1 librados: starting msgr at 2024-03-25T14:19:53.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 1 librados: starting objecter 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 1 Processor -- start 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 1 -- start start 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 1 librados: setting wanted keys 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 1 librados: calling monclient init 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.115+0000 7f8bd1881640 10 monclient: init 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 10 monclient(hunting): picked mon.noname-a con 0x7f8bcc0a96b0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:53.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc0b4700 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 10 monclient(hunting): picked mon.noname-b con 0x7f8bcc142a20 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0ac830 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc13cc10 con 0x7f8bcc142a20 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc13b7f0 con 0x7f8bcc0b4700 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bd1881640 10 monclient(hunting): authenticate will time out at 1350.266166s 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bcaffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8bcc0a96b0 0x7f8bcc13c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60686/0 (socket says 172.21.15.153:60686) 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.116+0000 7f8bcaffd640 1 -- 172.21.15.153:0/4175865151 learned_addr learned my addr 172.21.15.153:0/4175865151 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.117+0000 7f8bb3fff640 10 client.?.objecter ms_handle_connect 0x7f8bcc0a96b0 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.117+0000 7f8bb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.117+0000 7f8bb3fff640 10 client.?.objecter ms_handle_connect 0x7f8bcc142a20 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.117+0000 7f8bb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.117+0000 7f8bb3fff640 10 client.?.objecter ms_handle_connect 0x7f8bcc0b4700 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.117+0000 7f8bb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040148466 0 0) 0x7f8bcc0ac830 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): my global_id is 4137 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8004eb0 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238526423 0 0) 0x7f8bcc13cc10 con 0x7f8bcc142a20 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): my global_id is 4138 2024-03-25T14:19:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc0ac830 con 0x7f8bcc142a20 2024-03-25T14:19:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347281834 0 0) 0x7f8bcc13b7f0 con 0x7f8bcc0b4700 2024-03-25T14:19:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 10 monclient(hunting): my global_id is 4136 2024-03-25T14:19:53.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.119+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc13cc10 con 0x7f8bcc0b4700 2024-03-25T14:19:53.130 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:53.121+0000 7f3e02017040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:19:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3566950342 0 0) 0x7f8bcc0ac830 con 0x7f8bcc142a20 2024-03-25T14:19:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bcc13b7f0 con 0x7f8bcc142a20 2024-03-25T14:19:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8bc4001e80 con 0x7f8bcc142a20 2024-03-25T14:19:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:19:53.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 10 monclient(hunting): dump: 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:53.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 511534377 0 0) 0x7f8bb8004eb0 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bcc0ac830 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8bc0001a10 con 0x7f8bcc0a96b0 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.123+0000 7f8bb3fff640 10 monclient(hunting): dump: 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.134 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.125+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4009743141 0 0) 0x7f8bcc13cc10 con 0x7f8bcc0b4700 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.125+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bb8004eb0 con 0x7f8bcc0b4700 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.125+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8bbc0018d0 con 0x7f8bcc0b4700 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.125+0000 7f8bb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.125+0000 7f8bb3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.125+0000 7f8bb3fff640 10 monclient(hunting): dump: 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1767697038 0 0) 0x7f8bcc13b7f0 con 0x7f8bcc142a20 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 >> v1:172.21.15.63:6790/0 conn(0x7f8bcc0b4700 legacy=0x7f8bcc0acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 >> v1:172.21.15.63:6789/0 conn(0x7f8bcc0a96b0 legacy=0x7f8bcc13c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 10 monclient: _finish_hunting 0 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 1 monclient: found mon.b 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc13b980 con 0x7f8bcc142a20 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 10 monclient: _finish_auth 0 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 10 monclient: _check_auth_tickets 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 5 monclient: authenticate success, global_id 4138 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 10 monclient: _renew_subs 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bcc0ad360 con 0x7f8bcc142a20 2024-03-25T14:19:53.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 10 monclient: _renew_subs 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bcc13a520 con 0x7f8bcc142a20 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.127+0000 7f8bd1881640 1 librados: init done 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.128+0000 7f8bd1881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.128+0000 7f8bd1881640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.128+0000 7f8bd1881640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.128+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bcc0ac630 con 0x7f8bcc142a20 2024-03-25T14:19:53.138 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:53.137+0000 7ff6d0038040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T14:19:53.168 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:53.166+0000 7f3e02017040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T14:19:53.183 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:53.182+0000 7ff6d0038040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T14:19:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc4001fe0 con 0x7f8bcc142a20 2024-03-25T14:19:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:53.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8bc4004fb0 con 0x7f8bcc142a20 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 10 monclient: dump: 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8bc40054a0 con 0x7f8bcc142a20 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4117856402 0 0) 0x7f8bc4006a70 con 0x7f8bcc142a20 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 10 client.4138.objecter ms_dispatch 0x7f8bcc000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bc97fa640 4 set_mon_vals no callback set 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8bc4018de0 con 0x7f8bcc142a20 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.233+0000 7f8bb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:53.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.310+0000 7f8bd1881640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "07600efd-4d69-477c-a480-b036389aeb1a", "id": 3}] 2024-03-25T14:19:53.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.310+0000 7f8bd1881640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "07600efd-4d69-477c-a480-b036389aeb1a", "id": 3}] 2024-03-25T14:19:53.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.310+0000 7f8bd1881640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.310+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "07600efd-4d69-477c-a480-b036389aeb1a", "id": 3} v 0) v1 -- 0x7f8bcc04f730 con 0x7f8bcc142a20 2024-03-25T14:19:53.330 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:53.328+0000 7f3e02017040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T14:19:53.346 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:53.345+0000 7ff6d0038040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T14:19:53.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.395+0000 7f8bb3fff640 1 -- 172.21.15.153:0/4175865151 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07600efd-4d69-477c-a480-b036389aeb1a", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1850875253 0 525046306) 0x7f8bc4006710 con 0x7f8bcc142a20 2024-03-25T14:19:53.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.395+0000 7f8bb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "07600efd-4d69-477c-a480-b036389aeb1a", "id": 3}] 2024-03-25T14:19:53.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.395+0000 7f8bb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:53.397 INFO:teuthology.orchestra.run.smithi153.stdout:3 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 10 librados: watch_flush enter 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 10 librados: watch_flush exit 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 10 client.4138.objecter successfully canceled tick 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 10 monclient: shutdown 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 >> v1:172.21.15.153:6789/0 conn(0x7f8bcc142a20 legacy=0x7f8bcc1e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 shutdown_connections 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 >> 172.21.15.153:0/4175865151 conn(0x7f8bcc0668e0 msgr2=0x7f8bcc0b6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 shutdown_connections 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 1 -- 172.21.15.153:0/4175865151 wait complete. 2024-03-25T14:19:53.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.396+0000 7f8bd1881640 1 librados: shutdown 2024-03-25T14:19:53.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 3b31d7df-e7c2-4d5b-97b1-ae422be070d7 4 2024-03-25T14:19:53.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.507+0000 7f99e1baf640 10 monclient: get_monmap_and_config 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.507+0000 7f99e1baf640 10 monclient: build_initial_monmap 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.507+0000 7f99e1baf640 10 monclient: monmap: 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:53.508422+0000 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:53.508422+0000 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:53.510 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.508+0000 7f99e1baf640 1 Processor -- start 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.508+0000 7f99e1baf640 1 -- start start 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.508+0000 7f99e1baf640 10 monclient: init 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 10 monclient(hunting): picked mon.noname-c con 0x7f99dc0b4700 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 10 monclient(hunting): picked mon.noname-b con 0x7f99dc142a20 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 10 monclient(hunting): picked mon.noname-a con 0x7f99dc0a96b0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc0a9d60 con 0x7f99dc0a96b0 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc13ff10 con 0x7f99dc0b4700 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc0aa1f0 con 0x7f99dc142a20 2024-03-25T14:19:53.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:53.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99e1baf640 10 monclient(hunting): authenticate will time out at 1350.659005s 2024-03-25T14:19:53.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99dbfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f99dc0a96b0 0x7f99dc13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60692/0 (socket says 172.21.15.153:60692) 2024-03-25T14:19:53.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99db7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f99dc0b4700 0x7f99dc142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36880/0 (socket says 172.21.15.153:36880) 2024-03-25T14:19:53.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.509+0000 7f99dbfff640 1 -- 172.21.15.153:0/2378899692 learned_addr learned my addr 172.21.15.153:0/2378899692 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:53.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596508330 0 0) 0x7f99dc13ff10 con 0x7f99dc0b4700 2024-03-25T14:19:53.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): my global_id is 4141 2024-03-25T14:19:53.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c8003520 con 0x7f99dc0b4700 2024-03-25T14:19:53.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833095605 0 0) 0x7f99dc0a9d60 con 0x7f99dc0a96b0 2024-03-25T14:19:53.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 10 monclient(hunting): my global_id is 4140 2024-03-25T14:19:53.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.512+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99dc13ff10 con 0x7f99dc0a96b0 2024-03-25T14:19:53.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.513+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215985838 0 0) 0x7f99dc0aa1f0 con 0x7f99dc142a20 2024-03-25T14:19:53.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.513+0000 7f99da7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.513+0000 7f99da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.513+0000 7f99da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.513+0000 7f99da7fc640 10 monclient(hunting): my global_id is 4139 2024-03-25T14:19:53.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.513+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99dc0a9d60 con 0x7f99dc142a20 2024-03-25T14:19:53.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229704970 0 0) 0x7f99c8003520 con 0x7f99dc0b4700 2024-03-25T14:19:53.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 >> v1:172.21.15.63:6790/0 conn(0x7f99dc142a20 legacy=0x7f99dc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 >> v1:172.21.15.63:6789/0 conn(0x7f99dc0a96b0 legacy=0x7f99dc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 1 monclient: found mon.noname-c 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc0b4ae0 con 0x7f99dc0b4700 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 10 monclient: _finish_auth 0 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f99cc001a10 con 0x7f99dc0b4700 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99e1baf640 5 monclient: authenticate success, global_id 4141 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99e1baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99da7fc640 10 monclient: dump: 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.520 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.515+0000 7f99e1baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:53.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99cc001b70 con 0x7f99dc0b4700 2024-03-25T14:19:53.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99da7fc640 1 -- 172.21.15.153:0/2378899692 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f99cc001f80 con 0x7f99dc0b4700 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99da7fc640 10 monclient: dump: 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.523 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 4 set_mon_vals no callback set 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 10 monclient: shutdown 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 1 -- 172.21.15.153:0/2378899692 >> v1:172.21.15.153:6789/0 conn(0x7f99dc0b4700 legacy=0x7f99dc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 1 -- 172.21.15.153:0/2378899692 shutdown_connections 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 1 -- 172.21.15.153:0/2378899692 >> 172.21.15.153:0/2378899692 conn(0x7f99dc0668e0 msgr2=0x7f99dc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 1 -- 172.21.15.153:0/2378899692 shutdown_connections 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.517+0000 7f99e1baf640 1 -- 172.21.15.153:0/2378899692 wait complete. 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.518+0000 7f99e1baf640 10 monclient: build_initial_monmap 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.518+0000 7f99e1baf640 10 monclient: monmap: 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.518+0000 7f99e1baf640 1 librados: starting msgr at 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.518+0000 7f99e1baf640 1 librados: starting objecter 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 1 Processor -- start 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 1 -- start start 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 1 librados: setting wanted keys 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 1 librados: calling monclient init 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient: init 2024-03-25T14:19:53.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient(hunting): picked mon.noname-c con 0x7f99dc0a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient(hunting): picked mon.noname-b con 0x7f99dc0b4700 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient(hunting): picked mon.noname-a con 0x7f99dc142a20 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc0b1850 con 0x7f99dc142a20 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc0ad150 con 0x7f99dc0b4700 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc0ad430 con 0x7f99dc0a96b0 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.519+0000 7f99e1baf640 10 monclient(hunting): authenticate will time out at 1350.669401s 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99dbfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f99dc142a20 0x7f99dc1e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.153:60704/0 (socket says 172.21.15.153:60704) 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99dbfff640 1 -- 172.21.15.153:0/4148222803 learned_addr learned my addr 172.21.15.153:0/4148222803 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99d3fff640 10 client.?.objecter ms_handle_connect 0x7f99dc142a20 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99d3fff640 10 client.?.objecter ms_handle_connect 0x7f99dc0b4700 2024-03-25T14:19:53.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99d3fff640 10 client.?.objecter ms_handle_connect 0x7f99dc0a96b0 2024-03-25T14:19:53.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.520+0000 7f99d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875927634 0 0) 0x7f99dc0b1850 con 0x7f99dc142a20 2024-03-25T14:19:53.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): my global_id is 4143 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99bc004eb0 con 0x7f99dc142a20 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391170955 0 0) 0x7f99dc0ad150 con 0x7f99dc0b4700 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): my global_id is 4144 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99dc0b1850 con 0x7f99dc0b4700 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376970798 0 0) 0x7f99dc0ad430 con 0x7f99dc0a96b0 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 10 monclient(hunting): my global_id is 4142 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.522+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99dc0ad150 con 0x7f99dc0a96b0 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.526+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3226286831 0 0) 0x7f99dc0b1850 con 0x7f99dc0b4700 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.526+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99dc0ad430 con 0x7f99dc0b4700 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.526+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f99c4001870 con 0x7f99dc0b4700 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.526+0000 7f99d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.526+0000 7f99d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.526+0000 7f99d3fff640 10 monclient(hunting): dump: 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.531 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.526+0000 7f99d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2653796968 0 0) 0x7f99dc0ad150 con 0x7f99dc0a96b0 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99dc0b1850 con 0x7f99dc0a96b0 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f99cc001e80 con 0x7f99dc0a96b0 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:53.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 10 monclient(hunting): dump: 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3215204728 0 0) 0x7f99bc004eb0 con 0x7f99dc142a20 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99dc0ad150 con 0x7f99dc142a20 2024-03-25T14:19:53.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f99d40018f0 con 0x7f99dc142a20 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.527+0000 7f99d3fff640 10 monclient(hunting): dump: 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.534 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 460521662 0 0) 0x7f99dc0ad430 con 0x7f99dc0b4700 2024-03-25T14:19:53.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 >> v1:172.21.15.63:6790/0 conn(0x7f99dc0a96b0 legacy=0x7f99dc139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 >> v1:172.21.15.63:6789/0 conn(0x7f99dc142a20 legacy=0x7f99dc1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 10 monclient: _finish_hunting 0 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 1 monclient: found mon.b 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc0ac180 con 0x7f99dc0b4700 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 10 monclient: _finish_auth 0 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 10 monclient: _check_auth_tickets 2024-03-25T14:19:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 5 monclient: authenticate success, global_id 4144 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 10 monclient: _renew_subs 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99dc13b5c0 con 0x7f99dc0b4700 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 10 monclient: _renew_subs 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc0abaf0 con 0x7f99dc0b4700 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.530+0000 7f99e1baf640 1 librados: init done 2024-03-25T14:19:53.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.531+0000 7f99e1baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.531+0000 7f99e1baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.531+0000 7f99e1baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.531+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc04f730 con 0x7f99dc0b4700 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99c4003610 con 0x7f99dc0b4700 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f99c4001d80 con 0x7f99dc0b4700 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 10 monclient: dump: 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.539 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d9ffb640 4 set_mon_vals no callback set 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f99c4003d30 con 0x7f99dc0b4700 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 4266737174 0 0) 0x7f99c4005400 con 0x7f99dc0b4700 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 10 client.4144.objecter ms_dispatch 0x7f99dc000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T14:19:53.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.534+0000 7f99d3fff640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T14:19:53.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.629+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f99c4005000 con 0x7f99dc0b4700 2024-03-25T14:19:53.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.629+0000 7f99d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.629+0000 7f99d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:53.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.702+0000 7f99e1baf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3b31d7df-e7c2-4d5b-97b1-ae422be070d7", "id": 4}] 2024-03-25T14:19:53.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.702+0000 7f99e1baf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3b31d7df-e7c2-4d5b-97b1-ae422be070d7", "id": 4}] 2024-03-25T14:19:53.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.702+0000 7f99e1baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.702+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3b31d7df-e7c2-4d5b-97b1-ae422be070d7", "id": 4} v 0) v1 -- 0x7f99dc0b7190 con 0x7f99dc0b4700 2024-03-25T14:19:53.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.766+0000 7f99d3fff640 1 -- 172.21.15.153:0/4148222803 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b31d7df-e7c2-4d5b-97b1-ae422be070d7", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3926123731 0 1697012583) 0x7f99c4005610 con 0x7f99dc0b4700 2024-03-25T14:19:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.766+0000 7f99d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3b31d7df-e7c2-4d5b-97b1-ae422be070d7", "id": 4}] 2024-03-25T14:19:53.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.766+0000 7f99d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:53.768 INFO:teuthology.orchestra.run.smithi153.stdout:4 2024-03-25T14:19:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.766+0000 7f99e1baf640 10 librados: watch_flush enter 2024-03-25T14:19:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 10 librados: watch_flush exit 2024-03-25T14:19:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T14:19:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 10 client.4144.objecter successfully canceled tick 2024-03-25T14:19:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 10 monclient: shutdown 2024-03-25T14:19:53.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 >> v1:172.21.15.153:6789/0 conn(0x7f99dc0b4700 legacy=0x7f99dc0ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 shutdown_connections 2024-03-25T14:19:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 >> 172.21.15.153:0/4148222803 conn(0x7f99dc0668e0 msgr2=0x7f99dc1364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 shutdown_connections 2024-03-25T14:19:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 1 -- 172.21.15.153:0/4148222803 wait complete. 2024-03-25T14:19:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.767+0000 7f99e1baf640 1 librados: shutdown 2024-03-25T14:19:53.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 535c7ab0-1e04-488b-9b5b-c04f91b6ca9c 5 2024-03-25T14:19:53.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.878+0000 7f8f20d82640 10 monclient: get_monmap_and_config 2024-03-25T14:19:53.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.878+0000 7f8f20d82640 10 monclient: build_initial_monmap 2024-03-25T14:19:53.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.878+0000 7f8f20d82640 10 monclient: monmap: 2024-03-25T14:19:53.880 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:53.880 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.880 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:53.879007+0000 2024-03-25T14:19:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:53.879007+0000 2024-03-25T14:19:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:53.881 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 1 Processor -- start 2024-03-25T14:19:53.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 1 -- start start 2024-03-25T14:19:53.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient: init 2024-03-25T14:19:53.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:53.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:53.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c0b4e90 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c0b4340 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient(hunting): picked mon.noname-c con 0x7f8f1c13fd00 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0b5270 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0ab330 con 0x7f8f1c13fd00 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0a9d80 con 0x7f8f1c0b4340 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.879+0000 7f8f20d82640 10 monclient(hunting): authenticate will time out at 1351.029544s 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.880+0000 7f8f1ad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8f1c13fd00 0x7f8f1c146150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36896/0 (socket says 172.21.15.153:36896) 2024-03-25T14:19:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.880+0000 7f8f1ad9d640 1 -- 172.21.15.153:0/2202127028 learned_addr learned my addr 172.21.15.153:0/2202127028 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462821622 0 0) 0x7f8f1c0ab330 con 0x7f8f1c13fd00 2024-03-25T14:19:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): my global_id is 4147 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f08003520 con 0x7f8f1c13fd00 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191584426 0 0) 0x7f8f1c0b5270 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): my global_id is 4146 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c0ab330 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458838400 0 0) 0x7f8f1c0a9d80 con 0x7f8f1c0b4340 2024-03-25T14:19:53.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 10 monclient(hunting): my global_id is 4145 2024-03-25T14:19:53.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.882+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c0b5270 con 0x7f8f1c0b4340 2024-03-25T14:19:53.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2640956495 0 0) 0x7f8f08003520 con 0x7f8f1c13fd00 2024-03-25T14:19:53.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 >> v1:172.21.15.63:6790/0 conn(0x7f8f1c0b4340 legacy=0x7f8f1c145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 >> v1:172.21.15.63:6789/0 conn(0x7f8f1c0b4e90 legacy=0x7f8f1c13f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 10 monclient: _finish_hunting 0 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 1 monclient: found mon.noname-c 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0a96b0 con 0x7f8f1c13fd00 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 10 monclient: _finish_auth 0 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 10 monclient: _check_auth_tickets 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f14001a10 con 0x7f8f1c13fd00 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f20d82640 5 monclient: authenticate success, global_id 4147 2024-03-25T14:19:53.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f20d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f19d9b640 10 monclient: dump: 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.885+0000 7f8f20d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:53.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f14001b70 con 0x7f8f1c13fd00 2024-03-25T14:19:53.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f19d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:53.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f19d9b640 1 -- 172.21.15.153:0/2202127028 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f14001f80 con 0x7f8f1c13fd00 2024-03-25T14:19:53.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f19d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f19d9b640 10 monclient: dump: 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f20d82640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f20d82640 4 set_mon_vals no callback set 2024-03-25T14:19:53.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.887+0000 7f8f20d82640 10 monclient: shutdown 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.888+0000 7f8f20d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.888+0000 7f8f20d82640 1 -- 172.21.15.153:0/2202127028 >> v1:172.21.15.153:6789/0 conn(0x7f8f1c13fd00 legacy=0x7f8f1c146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.888+0000 7f8f20d82640 1 -- 172.21.15.153:0/2202127028 shutdown_connections 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.888+0000 7f8f20d82640 1 -- 172.21.15.153:0/2202127028 >> 172.21.15.153:0/2202127028 conn(0x7f8f1c066ac0 msgr2=0x7f8f1c066ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.888+0000 7f8f20d82640 1 -- 172.21.15.153:0/2202127028 shutdown_connections 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.888+0000 7f8f20d82640 1 -- 172.21.15.153:0/2202127028 wait complete. 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 10 monclient: build_initial_monmap 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 10 monclient: monmap: 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:53.893 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 1 librados: starting msgr at 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 1 librados: starting objecter 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 1 Processor -- start 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 1 -- start start 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 1 librados: setting wanted keys 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 1 librados: calling monclient init 2024-03-25T14:19:53.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.889+0000 7f8f20d82640 10 monclient: init 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 10 monclient(hunting): picked mon.noname-c con 0x7f8f1c0b4e90 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c13fd00 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c141710 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c134f80 con 0x7f8f1c13fd00 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c135260 con 0x7f8f1c141710 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c142bf0 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.890+0000 7f8f20d82640 10 monclient(hunting): authenticate will time out at 1351.040157s 2024-03-25T14:19:53.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f1ad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8f1c141710 0x7f8f1c1e25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36908/0 (socket says 172.21.15.153:36908) 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f1ad9d640 1 -- 172.21.15.153:0/514354526 learned_addr learned my addr 172.21.15.153:0/514354526 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f12ffd640 10 client.?.objecter ms_handle_connect 0x7f8f1c141710 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f12ffd640 10 client.?.objecter ms_handle_connect 0x7f8f1c13fd00 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f12ffd640 10 client.?.objecter ms_handle_connect 0x7f8f1c0b4e90 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.891+0000 7f8f12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930940318 0 0) 0x7f8f1c135260 con 0x7f8f1c141710 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T14:19:53.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc004eb0 con 0x7f8f1c141710 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835965831 0 0) 0x7f8f1c134f80 con 0x7f8f1c13fd00 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 10 monclient(hunting): my global_id is 4149 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c135260 con 0x7f8f1c13fd00 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.893+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990287916 0 0) 0x7f8f1c142bf0 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.894+0000 7f8f12ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.894+0000 7f8f12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.894+0000 7f8f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.894+0000 7f8f12ffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T14:19:53.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.894+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c134f80 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.898+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3606880285 0 0) 0x7f8f1c135260 con 0x7f8f1c13fd00 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.898+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c142bf0 con 0x7f8f1c13fd00 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.898+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f040018b0 con 0x7f8f1c13fd00 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.898+0000 7f8f12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.898+0000 7f8f12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.898+0000 7f8f12ffd640 10 monclient(hunting): dump: 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.901 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.901 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.898+0000 7f8f12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067537110 0 0) 0x7f8efc004eb0 con 0x7f8f1c141710 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c135260 con 0x7f8f1c141710 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1822955831 0 0) 0x7f8f1c134f80 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8efc004eb0 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f0c0018f0 con 0x7f8f1c0b4e90 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.899+0000 7f8f12ffd640 10 monclient(hunting): dump: 2024-03-25T14:19:53.902 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.900+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f14001f60 con 0x7f8f1c141710 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.900+0000 7f8f12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.900+0000 7f8f12ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.900+0000 7f8f12ffd640 10 monclient(hunting): dump: 2024-03-25T14:19:53.903 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 378209513 0 0) 0x7f8f1c142bf0 con 0x7f8f1c13fd00 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 >> v1:172.21.15.63:6790/0 conn(0x7f8f1c0b4e90 legacy=0x7f8f1c1359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 >> v1:172.21.15.153:6789/0 conn(0x7f8f1c141710 legacy=0x7f8f1c1e25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 1 monclient: found mon.a 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c1360e0 con 0x7f8f1c13fd00 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 10 monclient: _finish_auth 0 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 5 monclient: authenticate success, global_id 4149 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 10 monclient: _renew_subs 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c142880 con 0x7f8f1c13fd00 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 10 monclient: _renew_subs 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c134170 con 0x7f8f1c13fd00 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.902+0000 7f8f20d82640 1 librados: init done 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.903+0000 7f8f20d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.903+0000 7f8f20d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.903+0000 7f8f20d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:53.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.903+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c142d80 con 0x7f8f1c13fd00 2024-03-25T14:19:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f04003610 con 0x7f8f1c13fd00 2024-03-25T14:19:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f04003920 con 0x7f8f1c13fd00 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 10 monclient: dump: 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:53.909 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:53.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f1959a640 4 set_mon_vals no callback set 2024-03-25T14:19:53.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8f04003e10 con 0x7f8f1c13fd00 2024-03-25T14:19:53.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1582069931 0 0) 0x7f8f040054b0 con 0x7f8f1c13fd00 2024-03-25T14:19:53.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 10 client.4149.objecter ms_dispatch 0x7f8f1c000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T14:19:53.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T14:19:53.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T14:19:53.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:53.906+0000 7f8f12ffd640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T14:19:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.013+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8f04005040 con 0x7f8f1c13fd00 2024-03-25T14:19:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.013+0000 7f8f12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.013+0000 7f8f12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:54.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.086+0000 7f8f20d82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "535c7ab0-1e04-488b-9b5b-c04f91b6ca9c", "id": 5}] 2024-03-25T14:19:54.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.086+0000 7f8f20d82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "535c7ab0-1e04-488b-9b5b-c04f91b6ca9c", "id": 5}] 2024-03-25T14:19:54.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.086+0000 7f8f20d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:54.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.086+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd new", "uuid": "535c7ab0-1e04-488b-9b5b-c04f91b6ca9c", "id": 5} v 0) v1 -- 0x7f8f1c04f730 con 0x7f8f1c13fd00 2024-03-25T14:19:54.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.143+0000 7f8f12ffd640 1 -- 172.21.15.153:0/514354526 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "535c7ab0-1e04-488b-9b5b-c04f91b6ca9c", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2896818756 0 1988412176) 0x7f8f04005230 con 0x7f8f1c13fd00 2024-03-25T14:19:54.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.143+0000 7f8f12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "535c7ab0-1e04-488b-9b5b-c04f91b6ca9c", "id": 5}] 2024-03-25T14:19:54.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.143+0000 7f8f12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:54.145 INFO:teuthology.orchestra.run.smithi153.stdout:5 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 10 librados: watch_flush enter 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 10 librados: watch_flush exit 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 10 client.4149.objecter successfully canceled tick 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 10 monclient: shutdown 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 >> v1:172.21.15.63:6789/0 conn(0x7f8f1c13fd00 legacy=0x7f8f1c1deeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 shutdown_connections 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 >> 172.21.15.153:0/514354526 conn(0x7f8f1c066ac0 msgr2=0x7f8f1c0b3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 shutdown_connections 2024-03-25T14:19:54.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 1 -- 172.21.15.153:0/514354526 wait complete. 2024-03-25T14:19:54.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.144+0000 7f8f20d82640 1 librados: shutdown 2024-03-25T14:19:54.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 65bad5c1-0847-41d4-b4de-e5c8572569e4 6 2024-03-25T14:19:54.177 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:54.175+0000 7f3e02017040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T14:19:54.198 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:54.196+0000 7ff6d0038040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.256+0000 7fd2e87b6640 10 monclient: get_monmap_and_config 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.256+0000 7fd2e87b6640 10 monclient: build_initial_monmap 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.256+0000 7fd2e87b6640 10 monclient: monmap: 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:54.257228+0000 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:54.257228+0000 2024-03-25T14:19:54.259 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.257+0000 7fd2e87b6640 1 Processor -- start 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.257+0000 7fd2e87b6640 1 -- start start 2024-03-25T14:19:54.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.257+0000 7fd2e87b6640 10 monclient: init 2024-03-25T14:19:54.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:54.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:19:54.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e00b4700 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:54.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e0142a20 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:54.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e00a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:54.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00a9d60 con 0x7fd2e0142a20 2024-03-25T14:19:54.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e013ff10 con 0x7fd2e00b4700 2024-03-25T14:19:54.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00aa1f0 con 0x7fd2e00a96b0 2024-03-25T14:19:54.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:54.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e87b6640 10 monclient(hunting): authenticate will time out at 1351.407793s 2024-03-25T14:19:54.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e6552640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd2e00b4700 0x7fd2e0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36922/0 (socket says 172.21.15.153:36922) 2024-03-25T14:19:54.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.258+0000 7fd2e6552640 1 -- 172.21.15.153:0/4285688355 learned_addr learned my addr 172.21.15.153:0/4285688355 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.298+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331261190 0 0) 0x7fd2e013ff10 con 0x7fd2e00b4700 2024-03-25T14:19:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.298+0000 7fd2e5550640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.298+0000 7fd2e5550640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.298+0000 7fd2e5550640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.298+0000 7fd2e5550640 10 monclient(hunting): my global_id is 4153 2024-03-25T14:19:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.298+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2cc003520 con 0x7fd2e00b4700 2024-03-25T14:19:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.299+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945526432 0 0) 0x7fd2e00a9d60 con 0x7fd2e0142a20 2024-03-25T14:19:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.299+0000 7fd2e5550640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.299+0000 7fd2e5550640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.299+0000 7fd2e5550640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.299+0000 7fd2e5550640 10 monclient(hunting): my global_id is 4152 2024-03-25T14:19:54.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.299+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e013ff10 con 0x7fd2e0142a20 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515580706 0 0) 0x7fd2cc003520 con 0x7fd2e00b4700 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 >> v1:172.21.15.63:6790/0 conn(0x7fd2e00a96b0 legacy=0x7fd2e013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 >> v1:172.21.15.63:6789/0 conn(0x7fd2e0142a20 legacy=0x7fd2e013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 10 monclient: _finish_hunting 0 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 1 monclient: found mon.noname-c 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e00b4ae0 con 0x7fd2e00b4700 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 10 monclient: _finish_auth 0 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 10 monclient: _check_auth_tickets 2024-03-25T14:19:54.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e87b6640 5 monclient: authenticate success, global_id 4153 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e87b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd2d0001650 con 0x7fd2e00b4700 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.301+0000 7fd2e5550640 10 monclient: dump: 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.306 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.302+0000 7fd2e87b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2d00018a0 con 0x7fd2e00b4700 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e5550640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e5550640 1 -- 172.21.15.153:0/4285688355 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd2d0001c30 con 0x7fd2e00b4700 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e5550640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e5550640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e5550640 10 monclient: dump: 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.308 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e87b6640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.303+0000 7fd2e87b6640 4 set_mon_vals no callback set 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.304+0000 7fd2e87b6640 10 monclient: shutdown 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.304+0000 7fd2e87b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.304+0000 7fd2e87b6640 1 -- 172.21.15.153:0/4285688355 >> v1:172.21.15.153:6789/0 conn(0x7fd2e00b4700 legacy=0x7fd2e0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.304+0000 7fd2e87b6640 1 -- 172.21.15.153:0/4285688355 shutdown_connections 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.304+0000 7fd2e87b6640 1 -- 172.21.15.153:0/4285688355 >> 172.21.15.153:0/4285688355 conn(0x7fd2e00668e0 msgr2=0x7fd2e0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.304+0000 7fd2e87b6640 1 -- 172.21.15.153:0/4285688355 shutdown_connections 2024-03-25T14:19:54.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.304+0000 7fd2e87b6640 1 -- 172.21.15.153:0/4285688355 wait complete. 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.305+0000 7fd2e87b6640 10 monclient: build_initial_monmap 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.305+0000 7fd2e87b6640 10 monclient: monmap: 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.306+0000 7fd2e87b6640 1 librados: starting msgr at 2024-03-25T14:19:54.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.306+0000 7fd2e87b6640 1 librados: starting objecter 2024-03-25T14:19:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.306+0000 7fd2e87b6640 1 Processor -- start 2024-03-25T14:19:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.306+0000 7fd2e87b6640 1 -- start start 2024-03-25T14:19:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.306+0000 7fd2e87b6640 1 librados: setting wanted keys 2024-03-25T14:19:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.306+0000 7fd2e87b6640 1 librados: calling monclient init 2024-03-25T14:19:54.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.306+0000 7fd2e87b6640 10 monclient: init 2024-03-25T14:19:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:54.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e00a96b0 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e00b4700 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e0142a20 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00ad050 con 0x7fd2e0142a20 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00addb0 con 0x7fd2e00b4700 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e00ae020 con 0x7fd2e00a96b0 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e87b6640 10 monclient(hunting): authenticate will time out at 1351.457194s 2024-03-25T14:19:54.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e5d51640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd2e00b4700 0x7fd2e013a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36930/0 (socket says 172.21.15.153:36930) 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e6552640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fd2e00a96b0 0x7fd2e00ae790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.153:53546/0 (socket says 172.21.15.153:53546) 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.307+0000 7fd2e5d51640 1 -- 172.21.15.153:0/38739942 learned_addr learned my addr 172.21.15.153:0/38739942 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.308+0000 7fd2d6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2e00b4700 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.308+0000 7fd2d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.308+0000 7fd2d6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2e00a96b0 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.308+0000 7fd2d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.308+0000 7fd2d6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2e0142a20 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.308+0000 7fd2d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721677292 0 0) 0x7fd2e00addb0 con 0x7fd2e00b4700 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): my global_id is 4156 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4004eb0 con 0x7fd2e00b4700 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228407139 0 0) 0x7fd2e00ae020 con 0x7fd2e00a96b0 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 10 monclient(hunting): my global_id is 4154 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.310+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e00addb0 con 0x7fd2e00a96b0 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.311+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416439012 0 0) 0x7fd2e00ad050 con 0x7fd2e0142a20 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.311+0000 7fd2d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.311+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.311+0000 7fd2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.311+0000 7fd2d6ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T14:19:54.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.311+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e00ae020 con 0x7fd2e0142a20 2024-03-25T14:19:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.315+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 910433131 0 0) 0x7fd2c4004eb0 con 0x7fd2e00b4700 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.315+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2e00ad050 con 0x7fd2e00b4700 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.315+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd2dc001a10 con 0x7fd2e00b4700 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.315+0000 7fd2d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.315+0000 7fd2d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.315+0000 7fd2d6ffd640 10 monclient(hunting): dump: 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.318 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.315+0000 7fd2d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.316+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3653901855 0 0) 0x7fd2e00addb0 con 0x7fd2e00a96b0 2024-03-25T14:19:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.316+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c4004eb0 con 0x7fd2e00a96b0 2024-03-25T14:19:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd2d0001bb0 con 0x7fd2e00a96b0 2024-03-25T14:19:54.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 10 monclient(hunting): dump: 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745670092 0 0) 0x7fd2e00ae020 con 0x7fd2e0142a20 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2e00addb0 con 0x7fd2e0142a20 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd2d8001a60 con 0x7fd2e0142a20 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.317+0000 7fd2d6ffd640 10 monclient(hunting): dump: 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4275409313 0 0) 0x7fd2e00ad050 con 0x7fd2e00b4700 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 >> v1:172.21.15.63:6790/0 conn(0x7fd2e00a96b0 legacy=0x7fd2e00ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 >> v1:172.21.15.63:6789/0 conn(0x7fd2e0142a20 legacy=0x7fd2e00ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 1 monclient: found mon.b 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e00ae300 con 0x7fd2e00b4700 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 10 monclient: _finish_auth 0 2024-03-25T14:19:54.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 5 monclient: authenticate success, global_id 4156 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 10 monclient: _renew_subs 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e00abca0 con 0x7fd2e00b4700 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 10 monclient: _renew_subs 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e01e32a0 con 0x7fd2e00b4700 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.318+0000 7fd2e87b6640 1 librados: init done 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.319+0000 7fd2e87b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.319+0000 7fd2e87b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.319+0000 7fd2e87b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.319+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e004f730 con 0x7fd2e00b4700 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2dc001c50 con 0x7fd2e00b4700 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd2dc003a60 con 0x7fd2e00b4700 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 10 monclient: dump: 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.421 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd2dc004010 con 0x7fd2e00b4700 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2e4d4f640 4 set_mon_vals no callback set 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 614873022 0 0) 0x7fd2dc006890 con 0x7fd2e00b4700 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 10 client.4156.objecter ms_dispatch 0x7fd2e0000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T14:19:54.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T14:19:54.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T14:19:54.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd2dc006bd0 con 0x7fd2e00b4700 2024-03-25T14:19:54.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:54.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.418+0000 7fd2d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:54.469 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:54.467+0000 7f3e02017040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T14:19:54.493 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:54.491+0000 7ff6d0038040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T14:19:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.491+0000 7fd2e87b6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "65bad5c1-0847-41d4-b4de-e5c8572569e4", "id": 6}] 2024-03-25T14:19:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.491+0000 7fd2e87b6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "65bad5c1-0847-41d4-b4de-e5c8572569e4", "id": 6}] 2024-03-25T14:19:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.491+0000 7fd2e87b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.491+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "65bad5c1-0847-41d4-b4de-e5c8572569e4", "id": 6} v 0) v1 -- 0x7fd2e0143340 con 0x7fd2e00b4700 2024-03-25T14:19:54.523 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:54.521+0000 7f3e02017040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T14:19:54.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.551+0000 7fd2d6ffd640 1 -- 172.21.15.153:0/38739942 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "65bad5c1-0847-41d4-b4de-e5c8572569e4", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3015150860 0 1113811849) 0x7fd2dc0062a0 con 0x7fd2e00b4700 2024-03-25T14:19:54.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.551+0000 7fd2d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "65bad5c1-0847-41d4-b4de-e5c8572569e4", "id": 6}] 2024-03-25T14:19:54.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.551+0000 7fd2d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:54.636 INFO:teuthology.orchestra.run.smithi153.stdout:6 2024-03-25T14:19:54.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 10 librados: watch_flush enter 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 10 librados: watch_flush exit 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 10 client.4156.objecter successfully canceled tick 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 10 monclient: shutdown 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 >> v1:172.21.15.153:6789/0 conn(0x7fd2e00b4700 legacy=0x7fd2e013a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 shutdown_connections 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.552+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 >> 172.21.15.153:0/38739942 conn(0x7fd2e00668e0 msgr2=0x7fd2e00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.553+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 shutdown_connections 2024-03-25T14:19:54.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.553+0000 7fd2e87b6640 1 -- 172.21.15.153:0/38739942 wait complete. 2024-03-25T14:19:54.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.553+0000 7fd2e87b6640 1 librados: shutdown 2024-03-25T14:19:54.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new ce24bc86-436e-44aa-8020-5fa5e0250e2a 7 2024-03-25T14:19:54.639 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:54.560+0000 7ff6d0038040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T14:19:54.645 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:54.644+0000 7f3e02017040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T14:19:54.682 INFO:tasks.ceph.mgr.y.smithi063.stderr:2024-03-25T14:19:54.681+0000 7ff6d0038040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T14:19:54.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.755+0000 7f9155d4d640 10 monclient: get_monmap_and_config 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.755+0000 7f9155d4d640 10 monclient: build_initial_monmap 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.755+0000 7f9155d4d640 10 monclient: monmap: 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:19:54.755565+0000 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:19:54.755565+0000 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:54.757 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.755+0000 7f9155d4d640 1 Processor -- start 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.755+0000 7f9155d4d640 1 -- start start 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.755+0000 7f9155d4d640 10 monclient: init 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f91500a9920 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f91500b45a0 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f9150146450 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915006b100 con 0x7f91500a9920 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91500b2940 con 0x7f9150146450 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915013fec0 con 0x7f91500b45a0 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:54.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.756+0000 7f9155d4d640 10 monclient(hunting): authenticate will time out at 1351.906164s 2024-03-25T14:19:54.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.757+0000 7f914ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f9150146450 0x7f915014dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36942/0 (socket says 172.21.15.153:36942) 2024-03-25T14:19:54.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.757+0000 7f914ffff640 1 -- 172.21.15.153:0/513608995 learned_addr learned my addr 172.21.15.153:0/513608995 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:54.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419006313 0 0) 0x7f91500b2940 con 0x7f9150146450 2024-03-25T14:19:54.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): my global_id is 4159 2024-03-25T14:19:54.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913c003520 con 0x7f9150146450 2024-03-25T14:19:54.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773003441 0 0) 0x7f915013fec0 con 0x7f91500b45a0 2024-03-25T14:19:54.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 10 monclient(hunting): my global_id is 4157 2024-03-25T14:19:54.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.759+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91500b2940 con 0x7f91500b45a0 2024-03-25T14:19:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716879759 0 0) 0x7f913c003520 con 0x7f9150146450 2024-03-25T14:19:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 >> v1:172.21.15.63:6790/0 conn(0x7f91500b45a0 legacy=0x7f9150145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 >> v1:172.21.15.63:6789/0 conn(0x7f91500a9920 legacy=0x7f915013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:19:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 10 monclient: _finish_hunting 0 2024-03-25T14:19:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 1 monclient: found mon.noname-c 2024-03-25T14:19:54.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150140050 con 0x7f9150146450 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 10 monclient: _finish_auth 0 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 10 monclient: _check_auth_tickets 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9148001a10 con 0x7f9150146450 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f914effd640 10 monclient: dump: 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.765 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f9155d4d640 5 monclient: authenticate success, global_id 4159 2024-03-25T14:19:54.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.762+0000 7f9155d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9148001b70 con 0x7f9150146450 2024-03-25T14:19:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f914effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:54.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f914effd640 1 -- 172.21.15.153:0/513608995 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9148001de0 con 0x7f9150146450 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f914effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f914effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f914effd640 10 monclient: dump: 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.769 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 4 set_mon_vals no callback set 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 10 monclient: shutdown 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 1 -- 172.21.15.153:0/513608995 >> v1:172.21.15.153:6789/0 conn(0x7f9150146450 legacy=0x7f915014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 1 -- 172.21.15.153:0/513608995 shutdown_connections 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 1 -- 172.21.15.153:0/513608995 >> 172.21.15.153:0/513608995 conn(0x7f91500b7f40 msgr2=0x7f91500b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.764+0000 7f9155d4d640 1 -- 172.21.15.153:0/513608995 shutdown_connections 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.765+0000 7f9155d4d640 1 -- 172.21.15.153:0/513608995 wait complete. 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.765+0000 7f9155d4d640 10 monclient: build_initial_monmap 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.765+0000 7f9155d4d640 10 monclient: monmap: 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:54.770 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.765+0000 7f9155d4d640 1 librados: starting msgr at 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.765+0000 7f9155d4d640 1 librados: starting objecter 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 1 Processor -- start 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 1 -- start start 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 1 librados: setting wanted keys 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 1 librados: calling monclient init 2024-03-25T14:19:54.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient: init 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f91500a9920 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f91500b45a0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f9150146450 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150144a00 con 0x7f9150146450 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91500abe60 con 0x7f91500b45a0 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9150144390 con 0x7f91500a9920 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.766+0000 7f9155d4d640 10 monclient(hunting): authenticate will time out at 1351.916452s 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.767+0000 7f9146ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f91500b45a0 0x7f91501daff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36946/0 (socket says 172.21.15.153:36946) 2024-03-25T14:19:54.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.767+0000 7f9146ffd640 1 -- 172.21.15.153:0/4042479578 learned_addr learned my addr 172.21.15.153:0/4042479578 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T14:19:54.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.767+0000 7f914cff9640 10 client.?.objecter ms_handle_connect 0x7f91500b45a0 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.767+0000 7f914cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.767+0000 7f914cff9640 10 client.?.objecter ms_handle_connect 0x7f91500a9920 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.767+0000 7f914cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618013691 0 0) 0x7f91500abe60 con 0x7f91500b45a0 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): my global_id is 4162 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9130004eb0 con 0x7f91500b45a0 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426391125 0 0) 0x7f9150144390 con 0x7f91500a9920 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:19:54.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 10 monclient(hunting): my global_id is 4160 2024-03-25T14:19:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.769+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91500abe60 con 0x7f91500a9920 2024-03-25T14:19:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.773+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 548864588 0 0) 0x7f9130004eb0 con 0x7f91500b45a0 2024-03-25T14:19:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.773+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9150144390 con 0x7f91500b45a0 2024-03-25T14:19:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.773+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9138001a10 con 0x7f91500b45a0 2024-03-25T14:19:54.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.773+0000 7f914cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.773+0000 7f914cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.773+0000 7f914cff9640 10 monclient(hunting): dump: 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.773+0000 7f914cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.774+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884593703 0 0) 0x7f91500abe60 con 0x7f91500a9920 2024-03-25T14:19:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.774+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9130004eb0 con 0x7f91500a9920 2024-03-25T14:19:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.774+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9140001830 con 0x7f91500a9920 2024-03-25T14:19:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.774+0000 7f914cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.774+0000 7f914cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.774+0000 7f914cff9640 10 monclient(hunting): dump: 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.778 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 351437226 0 0) 0x7f9150144390 con 0x7f91500b45a0 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 >> v1:172.21.15.63:6790/0 conn(0x7f91500a9920 legacy=0x7f91501da8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 >> v1:172.21.15.63:6789/0 conn(0x7f9150146450 legacy=0x7f91501e1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 10 monclient: _finish_hunting 0 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 1 monclient: found mon.b 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150144520 con 0x7f91500b45a0 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 10 monclient: _finish_auth 0 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 10 monclient: _check_auth_tickets 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f914cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 5 monclient: authenticate success, global_id 4162 2024-03-25T14:19:54.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 10 monclient: _renew_subs 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9150143910 con 0x7f91500b45a0 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 10 monclient: _renew_subs 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9150143cb0 con 0x7f91500b45a0 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.777+0000 7f9155d4d640 1 librados: init done 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.778+0000 7f9155d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.778+0000 7f9155d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.778+0000 7f9155d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.778+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915004f730 con 0x7f91500b45a0 2024-03-25T14:19:54.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9138001c50 con 0x7f91500b45a0 2024-03-25T14:19:54.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:54.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9138003a60 con 0x7f91500b45a0 2024-03-25T14:19:54.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 10 monclient: dump: 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:19:54.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9138004010 con 0x7f91500b45a0 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1904686388 0 0) 0x7f91380068d0 con 0x7f91500b45a0 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 10 client.4162.objecter ms_dispatch 0x7f9150000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914e7fc640 4 set_mon_vals no callback set 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9138018de0 con 0x7f91500b45a0 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:54.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.876+0000 7f914cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:54.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.949+0000 7f9155d4d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ce24bc86-436e-44aa-8020-5fa5e0250e2a", "id": 7}] 2024-03-25T14:19:54.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.949+0000 7f9155d4d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ce24bc86-436e-44aa-8020-5fa5e0250e2a", "id": 7}] 2024-03-25T14:19:54.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.949+0000 7f9155d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:19:54.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.949+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ce24bc86-436e-44aa-8020-5fa5e0250e2a", "id": 7} v 0) v1 -- 0x7f915013b4d0 con 0x7f91500b45a0 2024-03-25T14:19:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:54.988+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 432632610 0 0) 0x7f9138006270 con 0x7f91500b45a0 2024-03-25T14:19:55.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.292+0000 7f914cff9640 1 -- 172.21.15.153:0/4042479578 <== mon.1 v1:172.21.15.153:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce24bc86-436e-44aa-8020-5fa5e0250e2a", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2881498408 0 1371665406) 0x7f91380163b0 con 0x7f91500b45a0 2024-03-25T14:19:55.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.292+0000 7f914cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ce24bc86-436e-44aa-8020-5fa5e0250e2a", "id": 7}] 2024-03-25T14:19:55.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.292+0000 7f914cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:55.294 INFO:teuthology.orchestra.run.smithi153.stdout:7 2024-03-25T14:19:55.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 10 librados: watch_flush enter 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 10 librados: watch_flush exit 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 10 client.4162.objecter successfully canceled tick 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 10 monclient: shutdown 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 >> v1:172.21.15.153:6789/0 conn(0x7f91500b45a0 legacy=0x7f91501daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 shutdown_connections 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 >> 172.21.15.153:0/4042479578 conn(0x7f91500b7f40 msgr2=0x7f9150139150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 shutdown_connections 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 1 -- 172.21.15.153:0/4042479578 wait complete. 2024-03-25T14:19:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T14:19:55.293+0000 7f9155d4d640 1 librados: shutdown 2024-03-25T14:19:55.306 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.306 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T14:19:55.306 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T14:19:55.309 INFO:tasks.ceph.osd.0:Started 2024-03-25T14:19:55.309 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.309 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T14:19:55.309 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T14:19:55.311 INFO:tasks.ceph.osd.1:Started 2024-03-25T14:19:55.311 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.311 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T14:19:55.311 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T14:19:55.314 INFO:tasks.ceph.osd.2:Started 2024-03-25T14:19:55.314 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.314 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T14:19:55.314 DEBUG:teuthology.orchestra.run.smithi063:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T14:19:55.316 INFO:tasks.ceph.osd.3:Started 2024-03-25T14:19:55.316 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.317 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T14:19:55.317 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T14:19:55.319 INFO:tasks.ceph.osd.4:Started 2024-03-25T14:19:55.319 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.319 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T14:19:55.319 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T14:19:55.321 INFO:tasks.ceph.osd.5:Started 2024-03-25T14:19:55.321 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.321 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T14:19:55.322 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T14:19:55.323 INFO:tasks.ceph.osd.6:Started 2024-03-25T14:19:55.324 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T14:19:55.324 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T14:19:55.324 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T14:19:55.326 INFO:tasks.ceph.osd.7:Started 2024-03-25T14:19:55.326 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T14:19:55.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.476+0000 7faad9788640 10 monclient: get_monmap_and_config 2024-03-25T14:19:55.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.476+0000 7faad9788640 10 monclient: build_initial_monmap 2024-03-25T14:19:55.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.476+0000 7faad9788640 10 monclient: monmap: 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:19:55.476266+0000 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:19:55.476266+0000 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:55.478 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:55.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.477+0000 7faad9788640 1 Processor -- start 2024-03-25T14:19:55.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.477+0000 7faad9788640 1 -- start start 2024-03-25T14:19:55.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.477+0000 7faad9788640 10 monclient: init 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 10 monclient(hunting): picked mon.noname-a con 0x7faad40b4700 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 10 monclient(hunting): picked mon.noname-b con 0x7faad4142a20 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 10 monclient(hunting): picked mon.noname-c con 0x7faad40a96b0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad40a9d60 con 0x7faad40b4700 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad413ff10 con 0x7faad40a96b0 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad40aa1f0 con 0x7faad4142a20 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:55.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.478+0000 7faad9788640 10 monclient(hunting): authenticate will time out at 1358.621850s 2024-03-25T14:19:55.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.483+0000 7faacbfff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7faad4142a20 0x7faad413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:59440/0 (socket says 172.21.15.63:59440) 2024-03-25T14:19:55.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.483+0000 7faacbfff640 1 -- 172.21.15.63:0/3997544449 learned_addr learned my addr 172.21.15.63:0/3997544449 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:19:55.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.506+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057147559 0 0) 0x7faad413ff10 con 0x7faad40a96b0 2024-03-25T14:19:55.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.506+0000 7faad27fc640 10 monclient(hunting): _negotiate 2024-03-25T14:19:55.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.506+0000 7faad27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:55.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.506+0000 7faad27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:55.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.506+0000 7faad27fc640 10 monclient(hunting): my global_id is 4168 2024-03-25T14:19:55.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.506+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab4003520 con 0x7faad40a96b0 2024-03-25T14:19:55.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2900333266 0 0) 0x7faab4003520 con 0x7faad40a96b0 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 >> v1:172.21.15.63:6790/0 conn(0x7faad4142a20 legacy=0x7faad413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 >> v1:172.21.15.63:6789/0 conn(0x7faad40b4700 legacy=0x7faad4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 10 monclient: _finish_hunting 0 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 1 monclient: found mon.noname-c 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad40b4ae0 con 0x7faad40a96b0 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 10 monclient: _finish_auth 0 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 10 monclient: _check_auth_tickets 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:55.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7faacc001a10 con 0x7faad40a96b0 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad27fc640 10 monclient: dump: 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad9788640 5 monclient: authenticate success, global_id 4168 2024-03-25T14:19:55.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.516+0000 7faad9788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faacc001be0 con 0x7faad40a96b0 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad27fc640 1 -- 172.21.15.63:0/3997544449 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7faacc001ff0 con 0x7faad40a96b0 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad9788640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad9788640 4 set_mon_vals no callback set 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.531+0000 7faad27fc640 10 monclient: dump: 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.533 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:55.534 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:55.534 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:55.534 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:55.534 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:55.534 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:55.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.532+0000 7faad9788640 10 monclient: shutdown 2024-03-25T14:19:55.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.532+0000 7faad9788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:55.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.532+0000 7faad9788640 1 -- 172.21.15.63:0/3997544449 >> v1:172.21.15.153:6789/0 conn(0x7faad40a96b0 legacy=0x7faad413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.532+0000 7faad9788640 1 -- 172.21.15.63:0/3997544449 shutdown_connections 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.532+0000 7faad9788640 1 -- 172.21.15.63:0/3997544449 >> 172.21.15.63:0/3997544449 conn(0x7faad40668e0 msgr2=0x7faad4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.532+0000 7faad9788640 1 -- 172.21.15.63:0/3997544449 shutdown_connections 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.532+0000 7faad9788640 1 -- 172.21.15.63:0/3997544449 wait complete. 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.533+0000 7faad9788640 10 monclient: build_initial_monmap 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.533+0000 7faad9788640 10 monclient: monmap: 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:55.536 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.533+0000 7faad9788640 1 librados: starting msgr at 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.533+0000 7faad9788640 1 librados: starting objecter 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.534+0000 7faad9788640 1 Processor -- start 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.534+0000 7faad9788640 1 -- start start 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.534+0000 7faad9788640 1 librados: setting wanted keys 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.534+0000 7faad9788640 1 librados: calling monclient init 2024-03-25T14:19:55.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.534+0000 7faad9788640 10 monclient: init 2024-03-25T14:19:55.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:55.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:19:55.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 10 monclient(hunting): picked mon.noname-b con 0x7faad40a96b0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 10 monclient(hunting): picked mon.noname-c con 0x7faad40b4700 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 10 monclient(hunting): picked mon.noname-a con 0x7faad4142a20 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad413a520 con 0x7faad4142a20 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad41391a0 con 0x7faad40a96b0 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad4138bc0 con 0x7faad40b4700 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.535+0000 7faad9788640 10 monclient(hunting): authenticate will time out at 1358.678948s 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.537+0000 7faad2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7faad40a96b0 0x7faad4130ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36378/0 (socket says 172.21.15.63:36378) 2024-03-25T14:19:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.537+0000 7faad2ffd640 1 -- 172.21.15.63:0/2236672390 learned_addr learned my addr 172.21.15.63:0/2236672390 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:19:55.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.538+0000 7faacb7fe640 10 client.?.objecter ms_handle_connect 0x7faad40a96b0 2024-03-25T14:19:55.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.538+0000 7faacb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:55.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.538+0000 7faacb7fe640 10 client.?.objecter ms_handle_connect 0x7faad40b4700 2024-03-25T14:19:55.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.538+0000 7faacb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:55.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.540+0000 7faacb7fe640 10 client.?.objecter ms_handle_connect 0x7faad4142a20 2024-03-25T14:19:55.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.540+0000 7faacb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:55.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.541+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991352360 0 0) 0x7faad4138bc0 con 0x7faad40b4700 2024-03-25T14:19:55.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.541+0000 7faacb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:19:55.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.541+0000 7faacb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:55.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.541+0000 7faacb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:55.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.541+0000 7faacb7fe640 10 monclient(hunting): my global_id is 4166 2024-03-25T14:19:55.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.541+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab8004eb0 con 0x7faad40b4700 2024-03-25T14:19:55.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.546+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578449019 0 0) 0x7faad41391a0 con 0x7faad40a96b0 2024-03-25T14:19:55.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.546+0000 7faacb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:19:55.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.546+0000 7faacb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:55.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.546+0000 7faacb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:55.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.546+0000 7faacb7fe640 10 monclient(hunting): my global_id is 4174 2024-03-25T14:19:55.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.546+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad4138bc0 con 0x7faad40a96b0 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.547+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496538285 0 0) 0x7faad413a520 con 0x7faad4142a20 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.547+0000 7faacb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 10 monclient(hunting): my global_id is 4164 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad41391a0 con 0x7faad4142a20 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1398241175 0 0) 0x7faab8004eb0 con 0x7faad40b4700 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faad413a520 con 0x7faad40b4700 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7faac4001800 con 0x7faad40b4700 2024-03-25T14:19:55.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 10 monclient(hunting): dump: 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:55.551 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:55.552 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:55.552 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:55.552 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:55.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.548+0000 7faacb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:55.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 732652018 0 0) 0x7faad413a520 con 0x7faad40b4700 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 >> v1:172.21.15.153:6789/0 conn(0x7faad40a96b0 legacy=0x7faad4130ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 >> v1:172.21.15.63:6789/0 conn(0x7faad4142a20 legacy=0x7faad41dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 1 monclient: found mon.c 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad4138d50 con 0x7faad40b4700 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 10 monclient: _finish_auth 0 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:19:55.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faacb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 5 monclient: authenticate success, global_id 4166 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 10 monclient: _renew_subs 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad41357f0 con 0x7faad40b4700 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 10 monclient: _renew_subs 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faad4138540 con 0x7faad40b4700 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.556+0000 7faad9788640 1 librados: init done 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.557+0000 7faad9788640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.558+0000 7faad9788640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.558+0000 7faad9788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:55.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.558+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad4138fa0 con 0x7faad40b4700 2024-03-25T14:19:55.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.559+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faac4001c70 con 0x7faad40b4700 2024-03-25T14:19:55.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.559+0000 7faacb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:55.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.559+0000 7faad1ffb640 4 set_mon_vals no callback set 2024-03-25T14:19:55.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.734+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7faac4004980 con 0x7faad40b4700 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.734+0000 7faacb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.734+0000 7faacb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.734+0000 7faacb7fe640 10 monclient: dump: 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:55.737 INFO:teuthology.orchestra.run.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:55.738 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:19:55.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.735+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 432632610 0 0) 0x7faac4004b60 con 0x7faad40b4700 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.738+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7faac40156f0 con 0x7faad40b4700 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.738+0000 7faacb7fe640 10 client.4166.objecter ms_dispatch 0x7faad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.738+0000 7faacb7fe640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.738+0000 7faacb7fe640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.738+0000 7faacb7fe640 20 client.4166.objecter dump_active .. 0 homeless 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.738+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faac4014b90 con 0x7faad40b4700 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.738+0000 7faacb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:55.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.739+0000 7faacb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:55.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.829+0000 7faad9788640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:19:55.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.829+0000 7faad9788640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:19:55.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.829+0000 7faad9788640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:19:55.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:55.829+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faad404f730 con 0x7faad40b4700 2024-03-25T14:19:56.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.051+0000 7faacb7fe640 1 -- 172.21.15.63:0/2236672390 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7faac4013190 con 0x7faad40b4700 2024-03-25T14:19:56.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.051+0000 7faacb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:19:56.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.051+0000 7faacb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:56.053 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-25T14:19:56.053 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:19:56.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.052+0000 7faad9788640 10 librados: watch_flush enter 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.052+0000 7faad9788640 10 librados: watch_flush exit 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.052+0000 7faad9788640 20 client.4166.objecter shutdown clearing up homeless session... 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.052+0000 7faad9788640 10 client.4166.objecter successfully canceled tick 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 10 monclient: shutdown 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 >> v1:172.21.15.63:6790/0 conn(0x7faad40b4700 legacy=0x7faad41da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 shutdown_connections 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 >> 172.21.15.63:0/2236672390 conn(0x7faad40668e0 msgr2=0x7faad413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 shutdown_connections 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 1 -- 172.21.15.63:0/2236672390 wait complete. 2024-03-25T14:19:56.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-25T14:19:56.053+0000 7faad9788640 1 librados: shutdown 2024-03-25T14:19:56.065 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T14:19:56.065 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T14:19:56.460 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:56.458+0000 7f3da2982640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T14:19:56.656 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:19:56.654+0000 7f3da2982640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T14:19:56.871 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.014+0000 7fd072523640 10 monclient: get_monmap_and_config 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.014+0000 7fd072523640 10 monclient: build_initial_monmap 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.014+0000 7fd072523640 10 monclient: monmap: 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:19:57.014730+0000 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:19:57.014730+0000 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:57.016 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:19:57.017 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:19:57.017 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:19:57.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.015+0000 7fd072523640 1 Processor -- start 2024-03-25T14:19:57.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.015+0000 7fd072523640 1 -- start start 2024-03-25T14:19:57.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.015+0000 7fd072523640 10 monclient: init 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.016+0000 7fd072523640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 10 monclient(hunting): picked mon.noname-c con 0x7fd06c0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 10 monclient(hunting): picked mon.noname-b con 0x7fd06c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 10 monclient(hunting): picked mon.noname-a con 0x7fd06c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c142520 con 0x7fd06c1430c0 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c0b28a0 con 0x7fd06c0a98f0 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c0b2a30 con 0x7fd06c13f710 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:57.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.017+0000 7fd072523640 10 monclient(hunting): authenticate will time out at 1360.161043s 2024-03-25T14:19:57.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.030+0000 7fd070ac0640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd06c1430c0 0x7fd06c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37578/0 (socket says 172.21.15.63:37578) 2024-03-25T14:19:57.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.030+0000 7fd070ac0640 1 -- 172.21.15.63:0/3533494014 learned_addr learned my addr 172.21.15.63:0/3533494014 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:19:57.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.035+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137821943 0 0) 0x7fd06c142520 con 0x7fd06c1430c0 2024-03-25T14:19:57.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.035+0000 7fd06affd640 10 monclient(hunting): _negotiate 2024-03-25T14:19:57.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.035+0000 7fd06affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:57.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.035+0000 7fd06affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:57.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.035+0000 7fd06affd640 10 monclient(hunting): my global_id is 4170 2024-03-25T14:19:57.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.035+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd054003520 con 0x7fd06c1430c0 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453452488 0 0) 0x7fd054003520 con 0x7fd06c1430c0 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 >> v1:172.21.15.63:6790/0 conn(0x7fd06c13f710 legacy=0x7fd06c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 >> v1:172.21.15.153:6789/0 conn(0x7fd06c0a98f0 legacy=0x7fd06c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 10 monclient: _finish_hunting 0 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 1 monclient: found mon.noname-a 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c0b44c0 con 0x7fd06c1430c0 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 10 monclient: _finish_auth 0 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 10 monclient: _check_auth_tickets 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:57.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0640019f0 con 0x7fd06c1430c0 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd06affd640 10 monclient: dump: 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd072523640 5 monclient: authenticate success, global_id 4170 2024-03-25T14:19:57.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.040+0000 7fd072523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:19:57.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd064001c50 con 0x7fd06c1430c0 2024-03-25T14:19:57.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd06affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:57.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd06affd640 1 -- 172.21.15.63:0/3533494014 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0640019f0 con 0x7fd06c1430c0 2024-03-25T14:19:57.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd06affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd06affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd06affd640 10 monclient: dump: 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:57.047 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd072523640 10 monclient: get_monmap_and_config success 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.045+0000 7fd072523640 4 set_mon_vals no callback set 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.046+0000 7fd072523640 10 monclient: shutdown 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.046+0000 7fd072523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.046+0000 7fd072523640 1 -- 172.21.15.63:0/3533494014 >> v1:172.21.15.63:6789/0 conn(0x7fd06c1430c0 legacy=0x7fd06c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.046+0000 7fd072523640 1 -- 172.21.15.63:0/3533494014 shutdown_connections 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.046+0000 7fd072523640 1 -- 172.21.15.63:0/3533494014 >> 172.21.15.63:0/3533494014 conn(0x7fd06c0b7ea0 msgr2=0x7fd06c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:57.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.046+0000 7fd072523640 1 -- 172.21.15.63:0/3533494014 shutdown_connections 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.047+0000 7fd072523640 1 -- 172.21.15.63:0/3533494014 wait complete. 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.048+0000 7fd072523640 10 monclient: build_initial_monmap 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.048+0000 7fd072523640 10 monclient: monmap: 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:19:57.049 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:19:57.050 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:19:57.050 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:19:57.050 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:19:57.050 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:19:57.050 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:19:57.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.048+0000 7fd072523640 1 librados: starting msgr at 2024-03-25T14:19:57.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.049+0000 7fd072523640 1 librados: starting objecter 2024-03-25T14:19:57.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.049+0000 7fd072523640 1 Processor -- start 2024-03-25T14:19:57.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.049+0000 7fd072523640 1 -- start start 2024-03-25T14:19:57.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.049+0000 7fd072523640 1 librados: setting wanted keys 2024-03-25T14:19:57.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.049+0000 7fd072523640 1 librados: calling monclient init 2024-03-25T14:19:57.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.049+0000 7fd072523640 10 monclient: init 2024-03-25T14:19:57.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.051+0000 7fd072523640 10 monclient: _reopen_session rank -1 2024-03-25T14:19:57.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.051+0000 7fd072523640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:19:57.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.051+0000 7fd072523640 10 monclient(hunting): picked mon.noname-a con 0x7fd06c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:19:57.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.051+0000 7fd072523640 10 monclient(hunting): picked mon.noname-c con 0x7fd06c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:19:57.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.051+0000 7fd072523640 10 monclient(hunting): picked mon.noname-b con 0x7fd06c1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:19:57.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.051+0000 7fd072523640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c1dac30 con 0x7fd06c0a98f0 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.051+0000 7fd072523640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c136be0 con 0x7fd06c1430c0 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.052+0000 7fd072523640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06c1db7d0 con 0x7fd06c13f710 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.052+0000 7fd072523640 10 monclient(hunting): _renew_subs 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.052+0000 7fd072523640 10 monclient(hunting): authenticate will time out at 1360.195662s 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.052+0000 7fd06bfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd06c0a98f0 0x7fd06c135520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37590/0 (socket says 172.21.15.63:37590) 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.052+0000 7fd06bfff640 1 -- 172.21.15.63:0/3936350618 learned_addr learned my addr 172.21.15.63:0/3936350618 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.052+0000 7fd068ff9640 10 client.?.objecter ms_handle_connect 0x7fd06c0a98f0 2024-03-25T14:19:57.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.052+0000 7fd068ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:19:57.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.056+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118279240 0 0) 0x7fd06c1dac30 con 0x7fd06c0a98f0 2024-03-25T14:19:57.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.056+0000 7fd068ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:19:57.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.056+0000 7fd068ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:19:57.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.056+0000 7fd068ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:19:57.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.056+0000 7fd068ff9640 10 monclient(hunting): my global_id is 4173 2024-03-25T14:19:57.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.056+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd058004eb0 con 0x7fd06c0a98f0 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.064+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293353764 0 0) 0x7fd058004eb0 con 0x7fd06c0a98f0 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.064+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd06c1dac30 con 0x7fd06c0a98f0 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.064+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd060001960 con 0x7fd06c0a98f0 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.064+0000 7fd068ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.064+0000 7fd068ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.064+0000 7fd068ff9640 10 monclient(hunting): dump: 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:57.066 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:19:57.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.064+0000 7fd068ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2252216648 0 0) 0x7fd06c1dac30 con 0x7fd06c0a98f0 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 >> v1:172.21.15.63:6790/0 conn(0x7fd06c13f710 legacy=0x7fd06c1db0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 >> v1:172.21.15.153:6789/0 conn(0x7fd06c1430c0 legacy=0x7fd06c130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 1 monclient: found mon.a 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:57.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c1db960 con 0x7fd06c0a98f0 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 10 monclient: _finish_auth 0 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd068ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 5 monclient: authenticate success, global_id 4173 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 10 monclient: _renew_subs 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c139d20 con 0x7fd06c0a98f0 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 10 monclient: _renew_subs 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c13c9f0 con 0x7fd06c0a98f0 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.070+0000 7fd072523640 1 librados: init done 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.071+0000 7fd072523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:57.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.071+0000 7fd072523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:57.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.071+0000 7fd072523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:57.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.071+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c1daa00 con 0x7fd06c0a98f0 2024-03-25T14:19:57.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.540+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd060001ac0 con 0x7fd06c0a98f0 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.540+0000 7fd068ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.540+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd060001d40 con 0x7fd06c0a98f0 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.540+0000 7fd068ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.540+0000 7fd068ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.540+0000 7fd068ff9640 10 monclient: dump: 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:19:57.542 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:19:57.543 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:19:57.543 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:19:57.543 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:19:57.543 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:19:57.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.541+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 382686572 0 0) 0x7fd060003920 con 0x7fd06c0a98f0 2024-03-25T14:19:57.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.540+0000 7fd06a7fc640 4 set_mon_vals no callback set 2024-03-25T14:19:57.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.542+0000 7fd06b7fe640 10 monclient: get_auth_request con 0x7fd05806da90 auth_method 0 2024-03-25T14:19:57.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.543+0000 7fd068ff9640 10 client.4173.objecter ms_handle_connect 0x7fd05806da90 2024-03-25T14:19:57.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fd060054650 con 0x7fd06c0a98f0 2024-03-25T14:19:57.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 10 client.4173.objecter ms_dispatch 0x7fd06c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:19:57.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:19:57.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:19:57.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T14:19:57.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd060050160 con 0x7fd06c0a98f0 2024-03-25T14:19:57.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:19:57.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.549+0000 7fd068ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:19:57.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.804+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd06004fc10 con 0x7fd06c0a98f0 2024-03-25T14:19:57.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.888+0000 7fd072523640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:19:57.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.888+0000 7fd072523640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:19:57.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.888+0000 7fd072523640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:19:57.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:57.888+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd06c1daa00 con 0x7fd06c0a98f0 2024-03-25T14:19:58.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.001+0000 7fd068ff9640 1 -- 172.21.15.63:0/3936350618 <== mon.0 v1:172.21.15.63:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fd06c1daa00 con 0x7fd06c0a98f0 2024-03-25T14:19:58.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.001+0000 7fd068ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:19:58.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.001+0000 7fd068ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:19:58.003 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:19:58.003 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:19:58.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 10 librados: watch_flush enter 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 10 librados: watch_flush exit 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 10 client.4173.objecter successfully canceled tick 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd05806da90 legacy=0x7fd05806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 10 monclient: shutdown 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 >> v1:172.21.15.63:6789/0 conn(0x7fd06c0a98f0 legacy=0x7fd06c135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 shutdown_connections 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.004+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 >> 172.21.15.63:0/3936350618 conn(0x7fd06c0b7ea0 msgr2=0x7fd06c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.005+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 shutdown_connections 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.005+0000 7fd072523640 1 -- 172.21.15.63:0/3936350618 wait complete. 2024-03-25T14:19:58.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:19:58.005+0000 7fd072523640 1 librados: shutdown 2024-03-25T14:19:58.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:02.088 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:06.703 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.158 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:06.774 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.171 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:06.779 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.232 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:06.840 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.263 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:06.879 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.323 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:06.939 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.387 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:07.000 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.446 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:07.059 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.663 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:07.258 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.703 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T14:20:02.693+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:02.724 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:07.319 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.788 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T14:20:02.780+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:02.854 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:07.460 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.862 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:07.455 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.892 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T14:20:02.884+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:02.916 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:07.523 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:02.924 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:07.518 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.015 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T14:20:03.007+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:03.022 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:07.637 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.082 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:07.698 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.153 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:07.768 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.250 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:07.858 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.271 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-03-25T14:20:03.262+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:03.367 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:07.983 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.491 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:08.105 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.501 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-03-25T14:20:03.491+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:03.509 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-03-25T14:20:03.500+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:03.527 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:08.143 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.622 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:08.230 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.673 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-03-25T14:20:03.664+0000 5dc3480 -1 Falling back to public interface 2024-03-25T14:20:03.735 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:08.350 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.748 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:08.342 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.839 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:08.455 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.862 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:08.476 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:03.935 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:08.543 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.012 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:08.619 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.024 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:08.617 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.046 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:08.662 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.104 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:08.720 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.126 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:08.721 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.154 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:08.770 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.172 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:08.786 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.204 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:08.812 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.311 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:08.927 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.377 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:08.993 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.380 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:08.987 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.408 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:09.001 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.437 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:09.051 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.468 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.076 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.493 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.088 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.531 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:09.146 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.575 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.191 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.649 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:09.264 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.702 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:09.316 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.725 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:09.318 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.734 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.342 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.737 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:09.344 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.783 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.378 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.830 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:04.848 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.464 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.909 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:09.524 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.918 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:09.534 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.920+0000 7fd0b3710640 10 monclient: get_monmap_and_config 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.920+0000 7fd0b3710640 10 monclient: build_initial_monmap 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.920+0000 7fd0b3710640 10 monclient: monmap: 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:04.921886+0000 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:04.921886+0000 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:04.923 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:04.924 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:04.924 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:04.924 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.921+0000 7fd0b3710640 1 Processor -- start 2024-03-25T14:20:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.921+0000 7fd0b3710640 1 -- start start 2024-03-25T14:20:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.921+0000 7fd0b3710640 10 monclient: init 2024-03-25T14:20:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:20:04.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ac0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 10 monclient(hunting): picked mon.noname-c con 0x7fd0ac13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ac1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac142520 con 0x7fd0ac1430c0 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac0b28a0 con 0x7fd0ac13f710 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac0b2a30 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b3710640 10 monclient(hunting): authenticate will time out at 1368.066749s 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b1cad640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd0ac1430c0 0x7fd0ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48626/0 (socket says 172.21.15.63:48626) 2024-03-25T14:20:04.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.922+0000 7fd0b1cad640 1 -- 172.21.15.63:0/1404077342 learned_addr learned my addr 172.21.15.63:0/1404077342 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519786528 0 0) 0x7fd0ac0b28a0 con 0x7fd0ac13f710 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): my global_id is 4213 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd098003520 con 0x7fd0ac13f710 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531291447 0 0) 0x7fd0ac142520 con 0x7fd0ac1430c0 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:04.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 10 monclient(hunting): my global_id is 4200 2024-03-25T14:20:04.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.925+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ac0b28a0 con 0x7fd0ac1430c0 2024-03-25T14:20:04.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.927+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512649140 0 0) 0x7fd0ac0b2a30 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.927+0000 7fd09ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:20:04.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.927+0000 7fd09ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:04.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.927+0000 7fd09ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:04.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.927+0000 7fd09ffff640 10 monclient(hunting): my global_id is 4199 2024-03-25T14:20:04.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.927+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ac142520 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677873359 0 0) 0x7fd098003520 con 0x7fd0ac13f710 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 >> v1:172.21.15.63:6790/0 conn(0x7fd0ac0a98f0 legacy=0x7fd0ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 >> v1:172.21.15.63:6789/0 conn(0x7fd0ac1430c0 legacy=0x7fd0ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 1 monclient: found mon.noname-c 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac0b44c0 con 0x7fd0ac13f710 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 10 monclient: _finish_auth 0 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0a0001690 con 0x7fd0ac13f710 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd0b3710640 5 monclient: authenticate success, global_id 4213 2024-03-25T14:20:04.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd0b3710640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd09ffff640 10 monclient: dump: 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:04.933 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:04.934 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:04.934 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:04.934 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.929+0000 7fd0b3710640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:04.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0a00017f0 con 0x7fd0ac13f710 2024-03-25T14:20:04.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd09ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd09ffff640 1 -- 172.21.15.63:0/1404077342 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0a0001c00 con 0x7fd0ac13f710 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd09ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd09ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd09ffff640 10 monclient: dump: 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:04.936 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd0b3710640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.930+0000 7fd0b3710640 4 set_mon_vals no callback set 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.931+0000 7fd0b3710640 10 monclient: shutdown 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.931+0000 7fd0b3710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.931+0000 7fd0b3710640 1 -- 172.21.15.63:0/1404077342 >> v1:172.21.15.153:6789/0 conn(0x7fd0ac13f710 legacy=0x7fd0ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.931+0000 7fd0b3710640 1 -- 172.21.15.63:0/1404077342 shutdown_connections 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.931+0000 7fd0b3710640 1 -- 172.21.15.63:0/1404077342 >> 172.21.15.63:0/1404077342 conn(0x7fd0ac0b7ea0 msgr2=0x7fd0ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.932+0000 7fd0b3710640 1 -- 172.21.15.63:0/1404077342 shutdown_connections 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.932+0000 7fd0b3710640 1 -- 172.21.15.63:0/1404077342 wait complete. 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.932+0000 7fd0b3710640 10 monclient: build_initial_monmap 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 10 monclient: monmap: 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:04.937 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 1 librados: starting msgr at 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 1 librados: starting objecter 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 1 Processor -- start 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 1 -- start start 2024-03-25T14:20:04.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 1 librados: setting wanted keys 2024-03-25T14:20:04.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 1 librados: calling monclient init 2024-03-25T14:20:04.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.933+0000 7fd0b3710640 10 monclient: init 2024-03-25T14:20:04.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ac0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ac13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 10 monclient(hunting): picked mon.noname-c con 0x7fd0ac1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac1d77a0 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac138e40 con 0x7fd0ac13f710 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ac0aba30 con 0x7fd0ac1430c0 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b3710640 10 monclient(hunting): authenticate will time out at 1368.078643s 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b1cad640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fd0ac1430c0 0x7fd0ac134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34008/0 (socket says 172.21.15.63:34008) 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd0b1cad640 1 -- 172.21.15.63:0/117201125 learned_addr learned my addr 172.21.15.63:0/117201125 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:04.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd09dffb640 10 client.?.objecter ms_handle_connect 0x7fd0ac1430c0 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.934+0000 7fd09dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.935+0000 7fd09dffb640 10 client.?.objecter ms_handle_connect 0x7fd0ac0a98f0 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.935+0000 7fd09dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.935+0000 7fd09dffb640 10 client.?.objecter ms_handle_connect 0x7fd0ac13f710 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.935+0000 7fd09dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823224928 0 0) 0x7fd0ac1d77a0 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): my global_id is 4203 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd090004eb0 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678287495 0 0) 0x7fd0ac0aba30 con 0x7fd0ac1430c0 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:04.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): my global_id is 4202 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ac1d77a0 con 0x7fd0ac1430c0 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983376610 0 0) 0x7fd0ac138e40 con 0x7fd0ac13f710 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T14:20:04.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.937+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ac0aba30 con 0x7fd0ac13f710 2024-03-25T14:20:04.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.941+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3321922070 0 0) 0x7fd090004eb0 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.941+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0ac138e40 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418079857 0 0) 0x7fd0ac0aba30 con 0x7fd0ac13f710 2024-03-25T14:20:04.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd090004eb0 con 0x7fd0ac13f710 2024-03-25T14:20:04.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0a0001b00 con 0x7fd0ac13f710 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): dump: 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3002585940 0 0) 0x7fd0ac1d77a0 con 0x7fd0ac1430c0 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0ac0aba30 con 0x7fd0ac1430c0 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0a4001bd0 con 0x7fd0ac1430c0 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): dump: 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:04.947 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0a80019b0 con 0x7fd0ac0a98f0 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.942+0000 7fd09dffb640 10 monclient(hunting): dump: 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:04.948 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:04.949 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:04.949 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1062021290 0 0) 0x7fd090004eb0 con 0x7fd0ac13f710 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 >> v1:172.21.15.63:6790/0 conn(0x7fd0ac1430c0 legacy=0x7fd0ac134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 >> v1:172.21.15.63:6789/0 conn(0x7fd0ac0a98f0 legacy=0x7fd0ac138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 10 monclient: _finish_hunting 0 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 1 monclient: found mon.b 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac1d8340 con 0x7fd0ac13f710 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 10 monclient: _finish_auth 0 2024-03-25T14:20:04.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 10 monclient: _check_auth_tickets 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd09dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd0b3710640 5 monclient: authenticate success, global_id 4216 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd0b3710640 10 monclient: _renew_subs 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd0b3710640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.945+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ac134820 con 0x7fd0ac13f710 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.946+0000 7fd0b3710640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.946+0000 7fd0b3710640 10 monclient: _renew_subs 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.946+0000 7fd0b3710640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.946+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ac1380c0 con 0x7fd0ac13f710 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.946+0000 7fd0b3710640 1 librados: init done 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.947+0000 7fd0b3710640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.947+0000 7fd0b3710640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.947+0000 7fd0b3710640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:04.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.947+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ac0627b0 con 0x7fd0ac13f710 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.961+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0a0001c60 con 0x7fd0ac13f710 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.961+0000 7fd09dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.961+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd0a0003a40 con 0x7fd0ac13f710 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.961+0000 7fd09dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.961+0000 7fd09dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.961+0000 7fd09dffb640 10 monclient: dump: 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:04.964 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:04.965 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:04.965 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:04.965 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:04.965 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:04.965 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.961+0000 7fd09f7fe640 4 set_mon_vals no callback set 2024-03-25T14:20:04.965 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.962+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd0a0019a10 con 0x7fd0ac13f710 2024-03-25T14:20:04.965 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.963+0000 7fd0b14ac640 10 monclient: get_auth_request con 0x7fd09006e080 auth_method 0 2024-03-25T14:20:04.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.963+0000 7fd09dffb640 10 client.4216.objecter ms_handle_connect 0x7fd09006e080 2024-03-25T14:20:04.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.964+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fd0a0053560 con 0x7fd0ac13f710 2024-03-25T14:20:04.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.964+0000 7fd09dffb640 10 client.4216.objecter ms_dispatch 0x7fd0ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:04.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.964+0000 7fd09dffb640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:04.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.964+0000 7fd09dffb640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:04.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:04.964+0000 7fd09dffb640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T14:20:04.969 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:09.583 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:04.994 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:09.587 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.004 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.612 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.028 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:09.634 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.072 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.667 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.113 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.729 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.183 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:09.799 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.199+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0a004f070 con 0x7fd0ac13f710 2024-03-25T14:20:05.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.199+0000 7fd09dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:05.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.199+0000 7fd09dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:05.204 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:09.820 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.234 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:09.848 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.258 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:09.851 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.272 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.880 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.317 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:09.924 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.373 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:09.967 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.377 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.993 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.397+0000 7fd0b3710640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:05.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.397+0000 7fd0b3710640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:05.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.397+0000 7fd0b3710640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:05.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.397+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0ac14e880 con 0x7fd0ac13f710 2024-03-25T14:20:05.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.443+0000 7fd09dffb640 1 -- 172.21.15.63:0/117201125 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fd0a0050fa0 con 0x7fd0ac13f710 2024-03-25T14:20:05.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.443+0000 7fd09dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:05.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.443+0000 7fd09dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:05.446 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:05.446 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:05.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 10 librados: watch_flush enter 2024-03-25T14:20:05.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 10 librados: watch_flush exit 2024-03-25T14:20:05.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T14:20:05.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 10 client.4216.objecter successfully canceled tick 2024-03-25T14:20:05.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd09006e080 legacy=0x7fd090070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:05.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 10 monclient: shutdown 2024-03-25T14:20:05.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:05.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 >> v1:172.21.15.153:6789/0 conn(0x7fd0ac13f710 legacy=0x7fd0ac1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:05.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 shutdown_connections 2024-03-25T14:20:05.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 >> 172.21.15.63:0/117201125 conn(0x7fd0ac0b7ea0 msgr2=0x7fd0ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:05.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 shutdown_connections 2024-03-25T14:20:05.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 1 -- 172.21.15.63:0/117201125 wait complete. 2024-03-25T14:20:05.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:05.445+0000 7fd0b3710640 1 librados: shutdown 2024-03-25T14:20:05.449 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:10.064 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:05.495 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:10.111 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.499 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:10.113 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.536 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:10.129 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.536 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:10.144 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.626 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:10.232 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.641 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:10.257 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.667 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:10.262 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.770 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:10.384 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.802 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:10.418 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.806 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:10.400 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.918 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:10.524 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.920 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:10.536 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.956 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:10.551 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:05.975 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:10.590 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.024 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:10.632 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.071 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:10.664 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.094 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:10.702 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.099 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:10.714 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.121 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:10.736 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.175 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:10.791 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.208 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:10.815 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.247 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:10.842 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.255 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:10.869 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.313 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:10.927 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.340 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:10.933 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.387 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:11.003 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.501 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:11.108 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.678 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:11.294 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.783 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:11.378 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.828 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:11.421 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.868 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:11.463 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:06.884 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:11.477 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:07.056 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:11.663 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:07.115 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:11.722 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:07.262 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:11.877 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:07.330 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:11.945 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:11.677 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:16.292 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:11.727 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:16.342 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:11.943 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:16.551 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:12.017 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:16.631 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:12.264 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:12.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.398+0000 7f7e6b425640 10 monclient: get_monmap_and_config 2024-03-25T14:20:12.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.398+0000 7f7e6b425640 10 monclient: build_initial_monmap 2024-03-25T14:20:12.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.398+0000 7f7e6b425640 10 monclient: monmap: 2024-03-25T14:20:12.401 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:12.401 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:12.401 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:12.399980+0000 2024-03-25T14:20:12.402 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:12.399980+0000 2024-03-25T14:20:12.402 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:12.402 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:12.402 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:12.402 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:12.402 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:12.402 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.399+0000 7f7e6b425640 1 Processor -- start 2024-03-25T14:20:12.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.399+0000 7f7e6b425640 1 -- start start 2024-03-25T14:20:12.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.399+0000 7f7e6b425640 10 monclient: init 2024-03-25T14:20:12.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:12.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 10 monclient(hunting): picked mon.noname-b con 0x7f7e640b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 10 monclient(hunting): picked mon.noname-a con 0x7f7e640b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 10 monclient(hunting): picked mon.noname-c con 0x7f7e6414be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e640af940 con 0x7f7e640b1730 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e640b0a70 con 0x7f7e6414be90 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e640b0d50 con 0x7f7e640b4d60 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e6b425640 10 monclient(hunting): authenticate will time out at 1375.544861s 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e689c0640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f7e640b1730 0x7f7e640b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:46246/0 (socket says 172.21.15.63:46246) 2024-03-25T14:20:12.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.400+0000 7f7e689c0640 1 -- 172.21.15.63:0/409207026 learned_addr learned my addr 172.21.15.63:0/409207026 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:12.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929939242 0 0) 0x7f7e640af940 con 0x7f7e640b1730 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): my global_id is 4206 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e50003520 con 0x7f7e640b1730 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564440937 0 0) 0x7f7e640b0d50 con 0x7f7e640b4d60 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): my global_id is 4205 2024-03-25T14:20:12.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e640af940 con 0x7f7e640b4d60 2024-03-25T14:20:12.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718614324 0 0) 0x7f7e640b0a70 con 0x7f7e6414be90 2024-03-25T14:20:12.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:20:12.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:12.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:12.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 10 monclient(hunting): my global_id is 4219 2024-03-25T14:20:12.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.404+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e500039d0 con 0x7f7e6414be90 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1380444690 0 0) 0x7f7e640af940 con 0x7f7e640b4d60 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 >> v1:172.21.15.153:6789/0 conn(0x7f7e6414be90 legacy=0x7f7e6414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 >> v1:172.21.15.63:6789/0 conn(0x7f7e640b1730 legacy=0x7f7e640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 1 monclient: found mon.noname-b 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e640b1030 con 0x7f7e640b4d60 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 10 monclient: _finish_auth 0 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f7e600015f0 con 0x7f7e640b4d60 2024-03-25T14:20:12.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e5bfff640 10 monclient: dump: 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e6b425640 5 monclient: authenticate success, global_id 4205 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.409+0000 7f7e6b425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e60001750 con 0x7f7e640b4d60 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e5bfff640 1 -- 172.21.15.63:0/409207026 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f7e60001ba0 con 0x7f7e640b4d60 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e6b425640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e6b425640 4 set_mon_vals no callback set 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:12.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.411+0000 7f7e5bfff640 10 monclient: dump: 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:12.415 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.412+0000 7f7e6b425640 10 monclient: shutdown 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.412+0000 7f7e6b425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.412+0000 7f7e6b425640 1 -- 172.21.15.63:0/409207026 >> v1:172.21.15.63:6790/0 conn(0x7f7e640b4d60 legacy=0x7f7e6413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.412+0000 7f7e6b425640 1 -- 172.21.15.63:0/409207026 shutdown_connections 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.412+0000 7f7e6b425640 1 -- 172.21.15.63:0/409207026 >> 172.21.15.63:0/409207026 conn(0x7f7e640617a0 msgr2=0x7f7e64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.413+0000 7f7e6b425640 1 -- 172.21.15.63:0/409207026 shutdown_connections 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.413+0000 7f7e6b425640 1 -- 172.21.15.63:0/409207026 wait complete. 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 10 monclient: build_initial_monmap 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 10 monclient: monmap: 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:12.417 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:12.418 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:12.418 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:12.418 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:12.418 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:12.418 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:12.418 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 1 librados: starting msgr at 2024-03-25T14:20:12.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 1 librados: starting objecter 2024-03-25T14:20:12.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 1 Processor -- start 2024-03-25T14:20:12.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 1 -- start start 2024-03-25T14:20:12.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 1 librados: setting wanted keys 2024-03-25T14:20:12.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 1 librados: calling monclient init 2024-03-25T14:20:12.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.414+0000 7f7e6b425640 10 monclient: init 2024-03-25T14:20:12.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:12.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:20:12.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 10 monclient(hunting): picked mon.noname-c con 0x7f7e640b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:12.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 10 monclient(hunting): picked mon.noname-b con 0x7f7e6414be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:12.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 10 monclient(hunting): picked mon.noname-a con 0x7f7e6413d790 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e641d7bd0 con 0x7f7e6413d790 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e641d78a0 con 0x7f7e6414be90 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e641d8770 con 0x7f7e640b1730 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e6b425640 10 monclient(hunting): authenticate will time out at 1375.559804s 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e691c1640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f7e640b1730 0x7f7e6413e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:42212/0 (socket says 172.21.15.63:42212) 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.415+0000 7f7e691c1640 1 -- 172.21.15.63:0/2519788579 learned_addr learned my addr 172.21.15.63:0/2519788579 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.416+0000 7f7e59ffb640 10 client.?.objecter ms_handle_connect 0x7f7e640b1730 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.416+0000 7f7e59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.416+0000 7f7e59ffb640 10 client.?.objecter ms_handle_connect 0x7f7e6414be90 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.416+0000 7f7e59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.416+0000 7f7e59ffb640 10 client.?.objecter ms_handle_connect 0x7f7e6413d790 2024-03-25T14:20:12.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.416+0000 7f7e59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.418+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803448107 0 0) 0x7f7e641d8770 con 0x7f7e640b1730 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.418+0000 7f7e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.418+0000 7f7e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.418+0000 7f7e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.418+0000 7f7e59ffb640 10 monclient(hunting): my global_id is 4208 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.418+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e48004eb0 con 0x7f7e640b1730 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.419+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942060291 0 0) 0x7f7e641d78a0 con 0x7f7e6414be90 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.419+0000 7f7e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.419+0000 7f7e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.419+0000 7f7e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.419+0000 7f7e59ffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.419+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e641d8770 con 0x7f7e6414be90 2024-03-25T14:20:12.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.420+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530733192 0 0) 0x7f7e641d7bd0 con 0x7f7e6413d790 2024-03-25T14:20:12.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.420+0000 7f7e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:12.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.420+0000 7f7e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:12.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.420+0000 7f7e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:12.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.420+0000 7f7e59ffb640 10 monclient(hunting): my global_id is 4209 2024-03-25T14:20:12.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.420+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e641d78a0 con 0x7f7e6413d790 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.424+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887711692 0 0) 0x7f7e48004eb0 con 0x7f7e640b1730 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.424+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e641d7bd0 con 0x7f7e640b1730 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.424+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f7e60001b20 con 0x7f7e640b1730 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.424+0000 7f7e59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.424+0000 7f7e59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.424+0000 7f7e59ffb640 10 monclient(hunting): dump: 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.427 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:12.428 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:12.428 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:12.428 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:12.428 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:12.428 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.424+0000 7f7e59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:12.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.426+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 162651485 0 0) 0x7f7e641d8770 con 0x7f7e6414be90 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.426+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e48005b20 con 0x7f7e6414be90 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.426+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f7e54001950 con 0x7f7e6414be90 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.426+0000 7f7e59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.426+0000 7f7e59ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.426+0000 7f7e59ffb640 10 monclient(hunting): dump: 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:12.429 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:12.430 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:12.430 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.427+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3713003222 0 0) 0x7f7e641d78a0 con 0x7f7e6413d790 2024-03-25T14:20:12.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.427+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e641d8770 con 0x7f7e6413d790 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.427+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f7e5c001b10 con 0x7f7e6413d790 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.427+0000 7f7e59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.427+0000 7f7e59ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.427+0000 7f7e59ffb640 10 monclient(hunting): dump: 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:12.431 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:12.432 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2611449364 0 0) 0x7f7e641d7bd0 con 0x7f7e640b1730 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 >> v1:172.21.15.153:6789/0 conn(0x7f7e6414be90 legacy=0x7f7e641d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 >> v1:172.21.15.63:6789/0 conn(0x7f7e6413d790 legacy=0x7f7e641e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 1 monclient: found mon.c 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64145820 con 0x7f7e640b1730 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 10 monclient: _finish_auth 0 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 5 monclient: authenticate success, global_id 4208 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 10 monclient: _renew_subs 2024-03-25T14:20:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e641e37f0 con 0x7f7e640b1730 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 10 monclient: _renew_subs 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e641e3b60 con 0x7f7e640b1730 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.431+0000 7f7e6b425640 1 librados: init done 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.433+0000 7f7e6b425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.433+0000 7f7e6b425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.433+0000 7f7e6b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:20:12.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.433+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e640b22c0 con 0x7f7e640b1730 2024-03-25T14:20:12.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.715+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e600038a0 con 0x7f7e640b1730 2024-03-25T14:20:12.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.715+0000 7f7e59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.715+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f7e60001b20 con 0x7f7e640b1730 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.715+0000 7f7e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.715+0000 7f7e59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.715+0000 7f7e59ffb640 10 monclient: dump: 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:12.719 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:12.720 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:12.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.716+0000 7f7e5b7fe640 4 set_mon_vals no callback set 2024-03-25T14:20:12.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.716+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f7e60003b10 con 0x7f7e640b1730 2024-03-25T14:20:12.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.717+0000 7f7e689c0640 10 monclient: get_auth_request con 0x7f7e4806e410 auth_method 0 2024-03-25T14:20:12.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.718+0000 7f7e59ffb640 10 client.4208.objecter ms_handle_connect 0x7f7e4806e410 2024-03-25T14:20:12.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f7e600544f0 con 0x7f7e640b1730 2024-03-25T14:20:12.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 10 client.4208.objecter ms_dispatch 0x7f7e64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:12.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:12.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:12.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T14:20:12.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e60050000 con 0x7f7e640b1730 2024-03-25T14:20:12.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:12.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.722+0000 7f7e59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:12.818 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:17.412 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:12.955 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:17.549 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:12.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.979+0000 7f7e6b425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:12.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.979+0000 7f7e6b425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:12.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.979+0000 7f7e6b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:20:12.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.979+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e640627b0 con 0x7f7e640b1730 2024-03-25T14:20:12.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.997+0000 7f7e59ffb640 1 -- 172.21.15.63:0/2519788579 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f7e60051f30 con 0x7f7e640b1730 2024-03-25T14:20:12.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.997+0000 7f7e59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:13.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:12.997+0000 7f7e59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:13.000 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:13.000 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:13.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 10 librados: watch_flush enter 2024-03-25T14:20:13.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 10 librados: watch_flush exit 2024-03-25T14:20:13.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T14:20:13.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 10 client.4208.objecter successfully canceled tick 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 >> v1:172.21.15.153:6800/1688255552 conn(0x7f7e4806e410 legacy=0x7f7e480708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 10 monclient: shutdown 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 >> v1:172.21.15.63:6790/0 conn(0x7f7e640b1730 legacy=0x7f7e6413e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 shutdown_connections 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 >> 172.21.15.63:0/2519788579 conn(0x7f7e640617a0 msgr2=0x7f7e640b64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 shutdown_connections 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 1 -- 172.21.15.63:0/2519788579 wait complete. 2024-03-25T14:20:13.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:13.000+0000 7f7e6b425640 1 librados: shutdown 2024-03-25T14:20:13.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:13.025 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:17.631 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:13.344 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:17.959 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:16.815 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:21.431 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:16.819 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:21.435 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:17.161 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:21.769 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:17.165 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:21.773 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:17.273 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:21.889 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:17.277 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:21.893 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:17.377 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:21.991 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:17.383 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:21.997 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:18.467 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:23.060 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:18.470 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:23.064 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:18.700 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:23.294 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:18.706 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:23.300 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:18.972 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:23.578 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:18.976 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:23.583 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:19.051 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:23.666 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:19.055 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:23.670 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:19.824 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:19.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.940+0000 7f8e564ae640 10 monclient: get_monmap_and_config 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.940+0000 7f8e564ae640 10 monclient: build_initial_monmap 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.940+0000 7f8e564ae640 10 monclient: monmap: 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:19.941247+0000 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:19.941247+0000 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:19.943 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.940+0000 7f8e564ae640 1 Processor -- start 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 1 -- start start 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 10 monclient: init 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 10 monclient(hunting): picked mon.noname-b con 0x7f8e500b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 10 monclient(hunting): picked mon.noname-c con 0x7f8e500b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 10 monclient(hunting): picked mon.noname-a con 0x7f8e5014be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e500af940 con 0x7f8e5014be90 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.941+0000 7f8e564ae640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e500b0a70 con 0x7f8e500b1730 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.942+0000 7f8e564ae640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e500b0d50 con 0x7f8e500b4d60 2024-03-25T14:20:19.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.942+0000 7f8e564ae640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:19.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.942+0000 7f8e564ae640 10 monclient(hunting): authenticate will time out at 1383.086576s 2024-03-25T14:20:19.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.942+0000 7f8e4ffff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f8e500b4d60 0x7f8e5013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:41446/0 (socket says 172.21.15.63:41446) 2024-03-25T14:20:19.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.942+0000 7f8e4ffff640 1 -- 172.21.15.63:0/3581425292 learned_addr learned my addr 172.21.15.63:0/3581425292 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:19.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.945+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920245259 0 0) 0x7f8e500b0d50 con 0x7f8e500b4d60 2024-03-25T14:20:19.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.945+0000 7f8e4effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:19.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.945+0000 7f8e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:19.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.945+0000 7f8e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:19.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.945+0000 7f8e4effd640 10 monclient(hunting): my global_id is 4211 2024-03-25T14:20:19.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.945+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38003520 con 0x7f8e500b4d60 2024-03-25T14:20:19.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355596710 0 0) 0x7f8e500b0a70 con 0x7f8e500b1730 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): my global_id is 4225 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e500b0d50 con 0x7f8e500b1730 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715707062 0 0) 0x7f8e500af940 con 0x7f8e5014be90 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 10 monclient(hunting): my global_id is 4212 2024-03-25T14:20:19.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.946+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e500b0a70 con 0x7f8e5014be90 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2363499137 0 0) 0x7f8e38003520 con 0x7f8e500b4d60 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 >> v1:172.21.15.153:6789/0 conn(0x7f8e500b1730 legacy=0x7f8e500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 >> v1:172.21.15.63:6789/0 conn(0x7f8e5014be90 legacy=0x7f8e5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 10 monclient: _finish_hunting 0 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 1 monclient: found mon.noname-b 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:20:19.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e500b1030 con 0x7f8e500b4d60 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 10 monclient: _finish_auth 0 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 10 monclient: _check_auth_tickets 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8e40001690 con 0x7f8e500b4d60 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e4effd640 10 monclient: dump: 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:19.952 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:19.953 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:19.953 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:19.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e564ae640 5 monclient: authenticate success, global_id 4211 2024-03-25T14:20:19.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.948+0000 7f8e564ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:19.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.950+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e400017f0 con 0x7f8e500b4d60 2024-03-25T14:20:19.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.950+0000 7f8e4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:19.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e564ae640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e564ae640 4 set_mon_vals no callback set 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e4effd640 1 -- 172.21.15.63:0/3581425292 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8e40001c00 con 0x7f8e500b4d60 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e4effd640 10 monclient: dump: 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:19.954 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:19.955 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e564ae640 10 monclient: shutdown 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e564ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e564ae640 1 -- 172.21.15.63:0/3581425292 >> v1:172.21.15.63:6790/0 conn(0x7f8e500b4d60 legacy=0x7f8e5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e564ae640 1 -- 172.21.15.63:0/3581425292 shutdown_connections 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.951+0000 7f8e564ae640 1 -- 172.21.15.63:0/3581425292 >> 172.21.15.63:0/3581425292 conn(0x7f8e500617a0 msgr2=0x7f8e50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.952+0000 7f8e564ae640 1 -- 172.21.15.63:0/3581425292 shutdown_connections 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.952+0000 7f8e564ae640 1 -- 172.21.15.63:0/3581425292 wait complete. 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.953+0000 7f8e564ae640 10 monclient: build_initial_monmap 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.953+0000 7f8e564ae640 10 monclient: monmap: 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:19.956 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:19.957 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:19.957 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:19.957 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:19.957 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:19.957 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:19.957 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:19.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.954+0000 7f8e564ae640 1 librados: starting msgr at 2024-03-25T14:20:19.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.954+0000 7f8e564ae640 1 librados: starting objecter 2024-03-25T14:20:19.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.954+0000 7f8e564ae640 1 Processor -- start 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.954+0000 7f8e564ae640 1 -- start start 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.954+0000 7f8e564ae640 1 librados: setting wanted keys 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.954+0000 7f8e564ae640 1 librados: calling monclient init 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.954+0000 7f8e564ae640 10 monclient: init 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 10 monclient(hunting): picked mon.noname-b con 0x7f8e500b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 10 monclient(hunting): picked mon.noname-a con 0x7f8e5014be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 10 monclient(hunting): picked mon.noname-c con 0x7f8e5012e600 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5012f5d0 con 0x7f8e5014be90 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e50141670 con 0x7f8e500b1730 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5012f210 con 0x7f8e5012e600 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:19.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.955+0000 7f8e564ae640 10 monclient(hunting): authenticate will time out at 1383.100101s 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8e500b1730 0x7f8e50141d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:35768/0 (socket says 172.21.15.63:35768) 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4ffff640 1 -- 172.21.15.63:0/1472835622 learned_addr learned my addr 172.21.15.63:0/1472835622 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4cff9640 10 client.?.objecter ms_handle_connect 0x7f8e5012e600 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4cff9640 10 client.?.objecter ms_handle_connect 0x7f8e500b1730 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4cff9640 10 client.?.objecter ms_handle_connect 0x7f8e5014be90 2024-03-25T14:20:19.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.956+0000 7f8e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699503227 0 0) 0x7f8e5012f5d0 con 0x7f8e5014be90 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): my global_id is 4215 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3c004eb0 con 0x7f8e5014be90 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134584017 0 0) 0x7f8e5012f210 con 0x7f8e5012e600 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): my global_id is 4214 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5012f5d0 con 0x7f8e5012e600 2024-03-25T14:20:19.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369682258 0 0) 0x7f8e50141670 con 0x7f8e500b1730 2024-03-25T14:20:19.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:20:19.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:19.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:19.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 10 monclient(hunting): my global_id is 4228 2024-03-25T14:20:19.964 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.959+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5012f210 con 0x7f8e500b1730 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428042139 0 0) 0x7f8e3c004eb0 con 0x7f8e5014be90 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e50141670 con 0x7f8e5014be90 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2004718611 0 0) 0x7f8e5012f210 con 0x7f8e500b1730 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e3c005a30 con 0x7f8e500b1730 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8e40001c00 con 0x7f8e500b1730 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): dump: 2024-03-25T14:20:19.970 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8e44001970 con 0x7f8e5014be90 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:19.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): dump: 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1702853554 0 0) 0x7f8e5012f5d0 con 0x7f8e5012e600 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e5012f210 con 0x7f8e5012e600 2024-03-25T14:20:19.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8e48001b50 con 0x7f8e5012e600 2024-03-25T14:20:19.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:19.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:19.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.966+0000 7f8e4cff9640 10 monclient(hunting): dump: 2024-03-25T14:20:19.973 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:19.973 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:19.973 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.974 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:19.974 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:19.974 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:19.974 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:19.974 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:19.975 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:19.975 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.969+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 929494461 0 0) 0x7f8e3c005a30 con 0x7f8e500b1730 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.969+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 >> v1:172.21.15.63:6790/0 conn(0x7f8e5012e600 legacy=0x7f8e5013e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.969+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 >> v1:172.21.15.63:6789/0 conn(0x7f8e5014be90 legacy=0x7f8e5014e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.969+0000 7f8e4cff9640 10 monclient: _finish_hunting 0 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.969+0000 7f8e4cff9640 1 monclient: found mon.b 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.969+0000 7f8e4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.969+0000 7f8e4cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5014ecc0 con 0x7f8e500b1730 2024-03-25T14:20:19.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e4cff9640 10 monclient: _finish_auth 0 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e4cff9640 10 monclient: _check_auth_tickets 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 5 monclient: authenticate success, global_id 4228 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 10 monclient: _renew_subs 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5014ef70 con 0x7f8e500b1730 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 10 monclient: _renew_subs 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e501e8120 con 0x7f8e500b1730 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.970+0000 7f8e564ae640 1 librados: init done 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.971+0000 7f8e564ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:19.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.971+0000 7f8e564ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:19.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.971+0000 7f8e564ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:19.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:19.971+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e500627b0 con 0x7f8e500b1730 2024-03-25T14:20:20.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.231+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e400010d0 con 0x7f8e500b1730 2024-03-25T14:20:20.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.231+0000 7f8e4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.231+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8e40004c30 con 0x7f8e500b1730 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.231+0000 7f8e4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.231+0000 7f8e4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.231+0000 7f8e4cff9640 10 monclient: dump: 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:20.235 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:20.236 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:20.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.231+0000 7f8e4e7fc640 4 set_mon_vals no callback set 2024-03-25T14:20:20.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.232+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f8e4001ac00 con 0x7f8e500b1730 2024-03-25T14:20:20.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.233+0000 7f8e4f7fe640 10 monclient: get_auth_request con 0x7f8e3c06e410 auth_method 0 2024-03-25T14:20:20.236 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.233+0000 7f8e4cff9640 10 client.4228.objecter ms_handle_connect 0x7f8e3c06e410 2024-03-25T14:20:20.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.239+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f8e40054790 con 0x7f8e500b1730 2024-03-25T14:20:20.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.239+0000 7f8e4cff9640 10 client.4228.objecter ms_dispatch 0x7f8e50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:20.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.239+0000 7f8e4cff9640 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:20.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.239+0000 7f8e4cff9640 3 client.4228.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:20.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.240+0000 7f8e4cff9640 20 client.4228.objecter dump_active .. 0 homeless 2024-03-25T14:20:20.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.241+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e400500d0 con 0x7f8e500b1730 2024-03-25T14:20:20.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.241+0000 7f8e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:20.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.241+0000 7f8e4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:20.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.465+0000 7f8e564ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:20.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.465+0000 7f8e564ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:20.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.465+0000 7f8e564ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:20.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.465+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e5012f3a0 con 0x7f8e500b1730 2024-03-25T14:20:20.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.490+0000 7f8e4cff9640 1 -- 172.21.15.63:0/1472835622 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f8e400521d0 con 0x7f8e500b1730 2024-03-25T14:20:20.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.490+0000 7f8e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:20.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.490+0000 7f8e4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:20.493 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:20.493 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:20.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 10 librados: watch_flush enter 2024-03-25T14:20:20.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 10 librados: watch_flush exit 2024-03-25T14:20:20.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 20 client.4228.objecter shutdown clearing up homeless session... 2024-03-25T14:20:20.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 10 client.4228.objecter successfully canceled tick 2024-03-25T14:20:20.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 >> v1:172.21.15.153:6800/1688255552 conn(0x7f8e3c06e410 legacy=0x7f8e3c0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 10 monclient: shutdown 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 >> v1:172.21.15.153:6789/0 conn(0x7f8e500b1730 legacy=0x7f8e50141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 shutdown_connections 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 >> 172.21.15.63:0/1472835622 conn(0x7f8e500617a0 msgr2=0x7f8e5013c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.492+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 shutdown_connections 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.493+0000 7f8e564ae640 1 -- 172.21.15.63:0/1472835622 wait complete. 2024-03-25T14:20:20.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:20.493+0000 7f8e564ae640 1 librados: shutdown 2024-03-25T14:20:20.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:21.271 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:25.886 35382== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:21.357 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:25.965 35384== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:21.889 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:26.504 35381== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:22.097 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:26.711 35383== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:22.812 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:00:27.405 35372== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:23.046 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:00:27.640 35373== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:23.375 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:00:27.990 35370== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:23.464 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:00:28.071 35371== Warning: unimplemented fcntl command: 1038 2024-03-25T14:20:27.313 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:27.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.404+0000 7f40b4996640 10 monclient: get_monmap_and_config 2024-03-25T14:20:27.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.404+0000 7f40b4996640 10 monclient: build_initial_monmap 2024-03-25T14:20:27.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.404+0000 7f40b4996640 10 monclient: monmap: 2024-03-25T14:20:27.407 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:27.407 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:27.407 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:27.405730+0000 2024-03-25T14:20:27.407 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:27.405730+0000 2024-03-25T14:20:27.408 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:27.408 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:27.408 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:27.408 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:27.408 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:27.408 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.405+0000 7f40b4996640 1 Processor -- start 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.405+0000 7f40b4996640 1 -- start start 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.405+0000 7f40b4996640 10 monclient: init 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 10 monclient(hunting): picked mon.noname-a con 0x7f40b0070410 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 10 monclient(hunting): picked mon.noname-c con 0x7f40b00707f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 10 monclient(hunting): picked mon.noname-b con 0x7f40b013db40 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00adea0 con 0x7f40b0070410 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b01445a0 con 0x7f40b00707f0 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00720a0 con 0x7f40b013db40 2024-03-25T14:20:27.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:27.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40b4996640 10 monclient(hunting): authenticate will time out at 1390.550860s 2024-03-25T14:20:27.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40aed9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f40b013db40 0x7f40b014ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:41466/0 (socket says 172.21.15.63:41466) 2024-03-25T14:20:27.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.406+0000 7f40aed9d640 1 -- 172.21.15.63:0/1537247909 learned_addr learned my addr 172.21.15.63:0/1537247909 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:27.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.409+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600642190 0 0) 0x7f40b00720a0 con 0x7f40b013db40 2024-03-25T14:20:27.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.409+0000 7f40ad59a640 10 monclient(hunting): _negotiate 2024-03-25T14:20:27.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.409+0000 7f40ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:27.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.409+0000 7f40ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:27.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.409+0000 7f40ad59a640 10 monclient(hunting): my global_id is 4217 2024-03-25T14:20:27.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.409+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4098003520 con 0x7f40b013db40 2024-03-25T14:20:27.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816967490 0 0) 0x7f40b01445a0 con 0x7f40b00707f0 2024-03-25T14:20:27.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): _negotiate 2024-03-25T14:20:27.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:27.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:27.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): my global_id is 4231 2024-03-25T14:20:27.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b00720a0 con 0x7f40b00707f0 2024-03-25T14:20:27.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519861340 0 0) 0x7f40b00adea0 con 0x7f40b0070410 2024-03-25T14:20:27.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): _negotiate 2024-03-25T14:20:27.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:27.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:27.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 10 monclient(hunting): my global_id is 4218 2024-03-25T14:20:27.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.411+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b01445a0 con 0x7f40b0070410 2024-03-25T14:20:27.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2328181890 0 0) 0x7f4098003520 con 0x7f40b013db40 2024-03-25T14:20:27.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 >> v1:172.21.15.153:6789/0 conn(0x7f40b00707f0 legacy=0x7f40b013d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:27.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 >> v1:172.21.15.63:6789/0 conn(0x7f40b0070410 legacy=0x7f40b0143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:27.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 10 monclient: _finish_hunting 0 2024-03-25T14:20:27.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 1 monclient: found mon.noname-b 2024-03-25T14:20:27.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0072380 con 0x7f40b013db40 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 10 monclient: _finish_auth 0 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 10 monclient: _check_auth_tickets 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f40a8001a10 con 0x7f40b013db40 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40ad59a640 10 monclient: dump: 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.418 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40b4996640 5 monclient: authenticate success, global_id 4217 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.413+0000 7f40b4996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.414+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a8001b70 con 0x7f40b013db40 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.414+0000 7f40ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.414+0000 7f40b4996640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:27.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.414+0000 7f40b4996640 4 set_mon_vals no callback set 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.415+0000 7f40ad59a640 1 -- 172.21.15.63:0/1537247909 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f40a8001f80 con 0x7f40b013db40 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.415+0000 7f40ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.415+0000 7f40ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.415+0000 7f40ad59a640 10 monclient: dump: 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.423 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.415+0000 7f40b4996640 10 monclient: shutdown 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.415+0000 7f40b4996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.415+0000 7f40b4996640 1 -- 172.21.15.63:0/1537247909 >> v1:172.21.15.63:6790/0 conn(0x7f40b013db40 legacy=0x7f40b014ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.416+0000 7f40b4996640 1 -- 172.21.15.63:0/1537247909 shutdown_connections 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.416+0000 7f40b4996640 1 -- 172.21.15.63:0/1537247909 >> 172.21.15.63:0/1537247909 conn(0x7f40b006b630 msgr2=0x7f40b00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.416+0000 7f40b4996640 1 -- 172.21.15.63:0/1537247909 shutdown_connections 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.416+0000 7f40b4996640 1 -- 172.21.15.63:0/1537247909 wait complete. 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 10 monclient: build_initial_monmap 2024-03-25T14:20:27.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 10 monclient: monmap: 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 1 librados: starting msgr at 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 1 librados: starting objecter 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 1 Processor -- start 2024-03-25T14:20:27.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 1 -- start start 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 1 librados: setting wanted keys 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 1 librados: calling monclient init 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.417+0000 7f40b4996640 10 monclient: init 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 10 monclient(hunting): picked mon.noname-b con 0x7f40b0070410 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 10 monclient(hunting): picked mon.noname-c con 0x7f40b00707f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 10 monclient(hunting): picked mon.noname-a con 0x7f40b013db40 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b01410a0 con 0x7f40b013db40 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b0141380 con 0x7f40b0070410 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b013f850 con 0x7f40b00707f0 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:27.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.418+0000 7f40b4996640 10 monclient(hunting): authenticate will time out at 1390.563125s 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f40add9b640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f40b00707f0 0x7f40b00aba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:41472/0 (socket says 172.21.15.63:41472) 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f40add9b640 1 -- 172.21.15.63:0/508572198 learned_addr learned my addr 172.21.15.63:0/508572198 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f409effd640 10 client.?.objecter ms_handle_connect 0x7f40b00707f0 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f409effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f409effd640 10 client.?.objecter ms_handle_connect 0x7f40b0070410 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f409effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f409effd640 10 client.?.objecter ms_handle_connect 0x7f40b013db40 2024-03-25T14:20:27.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.419+0000 7f409effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938484553 0 0) 0x7f40b01410a0 con 0x7f40b013db40 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): my global_id is 4221 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4090004eb0 con 0x7f40b013db40 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655554358 0 0) 0x7f40b013f850 con 0x7f40b00707f0 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:27.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): my global_id is 4220 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b01410a0 con 0x7f40b00707f0 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937394409 0 0) 0x7f40b0141380 con 0x7f40b0070410 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.422+0000 7f409effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.423+0000 7f409effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.423+0000 7f409effd640 10 monclient(hunting): my global_id is 4234 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.423+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b013f850 con 0x7f40b0070410 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.427+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173280035 0 0) 0x7f4090004eb0 con 0x7f40b013db40 2024-03-25T14:20:27.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.427+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40b0141380 con 0x7f40b013db40 2024-03-25T14:20:27.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.428+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f40a8001ea0 con 0x7f40b013db40 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.428+0000 7f409effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.428+0000 7f409effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.428+0000 7f409effd640 10 monclient(hunting): dump: 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:27.434 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.428+0000 7f409effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 230109770 0 0) 0x7f40b01410a0 con 0x7f40b00707f0 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4090004eb0 con 0x7f40b00707f0 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f40a40019f0 con 0x7f40b00707f0 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 10 monclient(hunting): dump: 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.435 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3140745326 0 0) 0x7f40b013f850 con 0x7f40b0070410 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40b01410a0 con 0x7f40b0070410 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.430+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f40a0001830 con 0x7f40b0070410 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.431+0000 7f409effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.431+0000 7f409effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.431+0000 7f409effd640 10 monclient(hunting): dump: 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:27.436 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.437 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.437 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:27.437 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:27.437 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:27.437 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:27.437 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:27.437 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.432+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1100050364 0 0) 0x7f40b0141380 con 0x7f40b013db40 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 >> v1:172.21.15.63:6790/0 conn(0x7f40b00707f0 legacy=0x7f40b00aba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 >> v1:172.21.15.153:6789/0 conn(0x7f40b0070410 legacy=0x7f40b013fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 10 monclient: _finish_hunting 0 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 1 monclient: found mon.a 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0138030 con 0x7f40b013db40 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 10 monclient: _finish_auth 0 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 10 monclient: _check_auth_tickets 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f409effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f40b4996640 5 monclient: authenticate success, global_id 4221 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f40b4996640 10 monclient: _renew_subs 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.433+0000 7f40b4996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:20:27.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.434+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b01382e0 con 0x7f40b013db40 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.434+0000 7f40b4996640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.434+0000 7f40b4996640 10 monclient: _renew_subs 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.434+0000 7f40b4996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.434+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b0137610 con 0x7f40b013db40 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.434+0000 7f40b4996640 1 librados: init done 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.435+0000 7f40b4996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.435+0000 7f40b4996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.435+0000 7f40b4996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:20:27.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.435+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b00627b0 con 0x7f40b013db40 2024-03-25T14:20:27.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.438+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a8002000 con 0x7f40b013db40 2024-03-25T14:20:27.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.438+0000 7f409effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:27.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.438+0000 7f40acd99640 4 set_mon_vals no callback set 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.453+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f40a8004fb0 con 0x7f40b013db40 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.453+0000 7f409effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.453+0000 7f409effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.453+0000 7f409effd640 10 monclient: dump: 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.457 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.454+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f40a801af80 con 0x7f40b013db40 2024-03-25T14:20:27.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.455+0000 7f40ae59c640 10 monclient: get_auth_request con 0x7f409006e0e0 auth_method 0 2024-03-25T14:20:27.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.456+0000 7f409effd640 10 client.4221.objecter ms_handle_connect 0x7f409006e0e0 2024-03-25T14:20:27.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.457+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f40a8054b00 con 0x7f40b013db40 2024-03-25T14:20:27.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.457+0000 7f409effd640 10 client.4221.objecter ms_dispatch 0x7f40b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:27.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.457+0000 7f409effd640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:27.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.457+0000 7f409effd640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:27.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.457+0000 7f409effd640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T14:20:27.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.748+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40a8050440 con 0x7f40b013db40 2024-03-25T14:20:27.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.748+0000 7f409effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:27.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:27.748+0000 7f409effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:28.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.052+0000 7f40b4996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:28.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.052+0000 7f40b4996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:28.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.052+0000 7f40b4996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:20:28.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.052+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40b0074d20 con 0x7f40b013db40 2024-03-25T14:20:28.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.077+0000 7f409effd640 1 -- 172.21.15.63:0/508572198 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f40a8052540 con 0x7f40b013db40 2024-03-25T14:20:28.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.078+0000 7f409effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:28.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.078+0000 7f409effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:28.080 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:28.081 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.080+0000 7f40b4996640 10 librados: watch_flush enter 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.080+0000 7f40b4996640 10 librados: watch_flush exit 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.080+0000 7f40b4996640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.080+0000 7f40b4996640 10 client.4221.objecter successfully canceled tick 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 >> v1:172.21.15.153:6800/1688255552 conn(0x7f409006e0e0 legacy=0x7f40900705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 10 monclient: shutdown 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 >> v1:172.21.15.63:6789/0 conn(0x7f40b013db40 legacy=0x7f40b013f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 shutdown_connections 2024-03-25T14:20:28.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 >> 172.21.15.63:0/508572198 conn(0x7f40b006b630 msgr2=0x7f40b013c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:28.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 shutdown_connections 2024-03-25T14:20:28.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 1 -- 172.21.15.63:0/508572198 wait complete. 2024-03-25T14:20:28.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:28.081+0000 7f40b4996640 1 librados: shutdown 2024-03-25T14:20:28.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:29.643 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T14:20:29.637+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T14:20:30.369 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T14:20:30.363+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T14:20:30.588 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T14:20:30.567+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T14:20:30.710 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T14:20:30.705+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T14:20:32.438 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-03-25T14:20:32.434+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T14:20:32.488 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-03-25T14:20:32.482+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T14:20:32.505 INFO:tasks.ceph.osd.3.smithi063.stderr:2024-03-25T14:20:32.498+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T14:20:32.766 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-03-25T14:20:32.760+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T14:20:34.905 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:35.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.024+0000 7f4365179640 10 monclient: get_monmap_and_config 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.024+0000 7f4365179640 10 monclient: build_initial_monmap 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.024+0000 7f4365179640 10 monclient: monmap: 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:35.025267+0000 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:35.025267+0000 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:35.027 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.024+0000 7f4365179640 1 Processor -- start 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.024+0000 7f4365179640 1 -- start start 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.024+0000 7f4365179640 10 monclient: init 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 10 monclient(hunting): picked mon.noname-c con 0x7f43600a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 10 monclient(hunting): picked mon.noname-a con 0x7f436013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 10 monclient(hunting): picked mon.noname-b con 0x7f43601430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4360142520 con 0x7f436013f710 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43600b28a0 con 0x7f43600a98f0 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43600b2a30 con 0x7f43601430c0 2024-03-25T14:20:35.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:35.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.025+0000 7f4365179640 10 monclient(hunting): authenticate will time out at 1398.170381s 2024-03-25T14:20:35.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.035+0000 7f435ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f43600a98f0 0x7f43600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:57830/0 (socket says 172.21.15.63:57830) 2024-03-25T14:20:35.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.035+0000 7f435ed9d640 1 -- 172.21.15.63:0/3136438183 learned_addr learned my addr 172.21.15.63:0/3136438183 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:35.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.057+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715482990 0 0) 0x7f43600b2a30 con 0x7f43601430c0 2024-03-25T14:20:35.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.057+0000 7f435dd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:20:35.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.057+0000 7f435dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.057+0000 7f435dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.057+0000 7f435dd9b640 10 monclient(hunting): my global_id is 4247 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.057+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348003520 con 0x7f43601430c0 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.057+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055816532 0 0) 0x7f4360142520 con 0x7f436013f710 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): my global_id is 4248 2024-03-25T14:20:35.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43600b2a30 con 0x7f436013f710 2024-03-25T14:20:35.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825860551 0 0) 0x7f43600b28a0 con 0x7f43600a98f0 2024-03-25T14:20:35.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:20:35.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:35.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:35.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 10 monclient(hunting): my global_id is 4261 2024-03-25T14:20:35.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.058+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4360142520 con 0x7f43600a98f0 2024-03-25T14:20:35.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595224602 0 0) 0x7f43600b2a30 con 0x7f436013f710 2024-03-25T14:20:35.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 >> v1:172.21.15.63:6790/0 conn(0x7f43601430c0 legacy=0x7f436014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:35.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 >> v1:172.21.15.153:6789/0 conn(0x7f43600a98f0 legacy=0x7f43600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:35.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 10 monclient: _finish_hunting 0 2024-03-25T14:20:35.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 1 monclient: found mon.noname-a 2024-03-25T14:20:35.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:35.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600b44c0 con 0x7f436013f710 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 10 monclient: _finish_auth 0 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 10 monclient: _check_auth_tickets 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4354001670 con 0x7f436013f710 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f4365179640 5 monclient: authenticate success, global_id 4248 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f4365179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f435dd9b640 10 monclient: dump: 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.073 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.074 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:35.074 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:35.074 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:35.074 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:35.074 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:35.074 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:35.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.069+0000 7f4365179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:35.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4354001890 con 0x7f436013f710 2024-03-25T14:20:35.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f435dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:35.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f435dd9b640 1 -- 172.21.15.63:0/3136438183 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4354001c20 con 0x7f436013f710 2024-03-25T14:20:35.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f4365179640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:35.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f4365179640 4 set_mon_vals no callback set 2024-03-25T14:20:35.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f435dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f435dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.095+0000 7f435dd9b640 10 monclient: dump: 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:35.099 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:35.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.096+0000 7f4365179640 10 monclient: shutdown 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.096+0000 7f4365179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.096+0000 7f4365179640 1 -- 172.21.15.63:0/3136438183 >> v1:172.21.15.63:6789/0 conn(0x7f436013f710 legacy=0x7f43601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.096+0000 7f4365179640 1 -- 172.21.15.63:0/3136438183 shutdown_connections 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.096+0000 7f4365179640 1 -- 172.21.15.63:0/3136438183 >> 172.21.15.63:0/3136438183 conn(0x7f43600b7ea0 msgr2=0x7f43600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.096+0000 7f4365179640 1 -- 172.21.15.63:0/3136438183 shutdown_connections 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.096+0000 7f4365179640 1 -- 172.21.15.63:0/3136438183 wait complete. 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.097+0000 7f4365179640 10 monclient: build_initial_monmap 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.097+0000 7f4365179640 10 monclient: monmap: 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:35.102 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 1 librados: starting msgr at 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 1 librados: starting objecter 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 1 Processor -- start 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 1 -- start start 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 1 librados: setting wanted keys 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 1 librados: calling monclient init 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 10 monclient: init 2024-03-25T14:20:35.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.098+0000 7f4365179640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 10 monclient(hunting): picked mon.noname-b con 0x7f43600a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 10 monclient(hunting): picked mon.noname-a con 0x7f436013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 10 monclient(hunting): picked mon.noname-c con 0x7f43601430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43601dad00 con 0x7f436013f710 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4360139a10 con 0x7f43600a98f0 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43601db8a0 con 0x7f43601430c0 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:35.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.099+0000 7f4365179640 10 monclient(hunting): authenticate will time out at 1398.243764s 2024-03-25T14:20:35.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.103+0000 7f435ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f43600a98f0 0x7f4360137b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:57860/0 (socket says 172.21.15.63:57860) 2024-03-25T14:20:35.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.103+0000 7f435ed9d640 1 -- 172.21.15.63:0/2159121245 learned_addr learned my addr 172.21.15.63:0/2159121245 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:35.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.104+0000 7f43477fe640 10 client.?.objecter ms_handle_connect 0x7f43600a98f0 2024-03-25T14:20:35.106 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.104+0000 7f43477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:35.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.110+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455797877 0 0) 0x7f4360139a10 con 0x7f43600a98f0 2024-03-25T14:20:35.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.110+0000 7f43477fe640 10 monclient(hunting): _negotiate 2024-03-25T14:20:35.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.110+0000 7f43477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:35.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.110+0000 7f43477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:35.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.110+0000 7f43477fe640 10 monclient(hunting): my global_id is 4264 2024-03-25T14:20:35.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.110+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f434c004eb0 con 0x7f43600a98f0 2024-03-25T14:20:35.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.111+0000 7f43477fe640 10 client.?.objecter ms_handle_connect 0x7f43601430c0 2024-03-25T14:20:35.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.111+0000 7f43477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:35.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.121+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449482320 0 0) 0x7f43601db8a0 con 0x7f43601430c0 2024-03-25T14:20:35.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.121+0000 7f43477fe640 10 monclient(hunting): _negotiate 2024-03-25T14:20:35.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.121+0000 7f43477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:35.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.121+0000 7f43477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:35.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.121+0000 7f43477fe640 10 monclient(hunting): my global_id is 4250 2024-03-25T14:20:35.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.121+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4360139a10 con 0x7f43601430c0 2024-03-25T14:20:35.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.122+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91585127 0 0) 0x7f434c004eb0 con 0x7f43600a98f0 2024-03-25T14:20:35.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.122+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43601db8a0 con 0x7f43600a98f0 2024-03-25T14:20:35.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.122+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4350001910 con 0x7f43600a98f0 2024-03-25T14:20:35.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.122+0000 7f43477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:35.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.122+0000 7f43477fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:35.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.122+0000 7f43477fe640 10 monclient(hunting): dump: 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:35.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.123+0000 7f43477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:35.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1948245955 0 0) 0x7f43601db8a0 con 0x7f43600a98f0 2024-03-25T14:20:35.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 >> v1:172.21.15.63:6790/0 conn(0x7f43601430c0 legacy=0x7f43601e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:35.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 >> v1:172.21.15.63:6789/0 conn(0x7f436013f710 legacy=0x7f43601db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:35.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 10 monclient: _finish_hunting 0 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 1 monclient: found mon.b 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4360135080 con 0x7f43600a98f0 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 10 monclient: _finish_auth 0 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 10 monclient: _check_auth_tickets 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f43477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 5 monclient: authenticate success, global_id 4264 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 10 monclient: _renew_subs 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4360135270 con 0x7f43600a98f0 2024-03-25T14:20:35.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:35.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 10 monclient: _renew_subs 2024-03-25T14:20:35.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:35.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4360138b00 con 0x7f43600a98f0 2024-03-25T14:20:35.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.127+0000 7f4365179640 1 librados: init done 2024-03-25T14:20:35.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.128+0000 7f4365179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:35.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.128+0000 7f4365179640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:35.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.128+0000 7f4365179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:35.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.128+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43601daa60 con 0x7f43600a98f0 2024-03-25T14:20:35.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.426+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43500047e0 con 0x7f43600a98f0 2024-03-25T14:20:35.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.426+0000 7f43477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:35.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.426+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4350004bf0 con 0x7f43600a98f0 2024-03-25T14:20:35.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.426+0000 7f43477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:35.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.426+0000 7f43477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.426+0000 7f43477fe640 10 monclient: dump: 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:35.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.426+0000 7f435d59a640 4 set_mon_vals no callback set 2024-03-25T14:20:35.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.427+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f4350004e10 con 0x7f43600a98f0 2024-03-25T14:20:35.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.428+0000 7f435e59c640 10 monclient: get_auth_request con 0x7f434c06dd30 auth_method 0 2024-03-25T14:20:35.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.428+0000 7f43477fe640 10 client.4264.objecter ms_handle_connect 0x7f434c06dd30 2024-03-25T14:20:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.433+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f43500548a0 con 0x7f43600a98f0 2024-03-25T14:20:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.433+0000 7f43477fe640 10 client.4264.objecter ms_dispatch 0x7f4360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.433+0000 7f43477fe640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.433+0000 7f43477fe640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:35.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.433+0000 7f43477fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T14:20:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.434+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43500503b0 con 0x7f43600a98f0 2024-03-25T14:20:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.434+0000 7f43477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:35.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.434+0000 7f43477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:35.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.654+0000 7f4365179640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:35.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.654+0000 7f4365179640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:35.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.654+0000 7f4365179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:35.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.654+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43600627b0 con 0x7f43600a98f0 2024-03-25T14:20:35.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.712+0000 7f43477fe640 1 -- 172.21.15.63:0/2159121245 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f43500522e0 con 0x7f43600a98f0 2024-03-25T14:20:35.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.712+0000 7f43477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:35.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.712+0000 7f43477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:35.716 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:35.716 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:35.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.715+0000 7f4365179640 10 librados: watch_flush enter 2024-03-25T14:20:35.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 10 librados: watch_flush exit 2024-03-25T14:20:35.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T14:20:35.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 10 client.4264.objecter successfully canceled tick 2024-03-25T14:20:35.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 >> v1:172.21.15.153:6800/1688255552 conn(0x7f434c06dd30 legacy=0x7f434c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:35.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 10 monclient: shutdown 2024-03-25T14:20:35.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:35.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 >> v1:172.21.15.153:6789/0 conn(0x7f43600a98f0 legacy=0x7f4360137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:35.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 shutdown_connections 2024-03-25T14:20:35.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 >> 172.21.15.63:0/2159121245 conn(0x7f43600b7ea0 msgr2=0x7f4360074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:35.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 shutdown_connections 2024-03-25T14:20:35.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 1 -- 172.21.15.63:0/2159121245 wait complete. 2024-03-25T14:20:35.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:35.716+0000 7f4365179640 1 librados: shutdown 2024-03-25T14:20:35.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:42.538 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:42.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.652+0000 7f459b972640 10 monclient: get_monmap_and_config 2024-03-25T14:20:42.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.652+0000 7f459b972640 10 monclient: build_initial_monmap 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.652+0000 7f459b972640 10 monclient: monmap: 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:42.653966+0000 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:42.653966+0000 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:42.656 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:42.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.653+0000 7f459b972640 1 Processor -- start 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.653+0000 7f459b972640 1 -- start start 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.653+0000 7f459b972640 10 monclient: init 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 10 monclient(hunting): picked mon.noname-c con 0x7f459406adf0 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 10 monclient(hunting): picked mon.noname-b con 0x7f45940aee10 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 10 monclient(hunting): picked mon.noname-a con 0x7f4594140860 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45940af690 con 0x7f4594140860 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45940b0c70 con 0x7f459406adf0 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45940b0f50 con 0x7f45940aee10 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:42.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.654+0000 7f459b972640 10 monclient(hunting): authenticate will time out at 1405.799311s 2024-03-25T14:20:42.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.667+0000 7f459970e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f459406adf0 0x7f459413d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:52244/0 (socket says 172.21.15.63:52244) 2024-03-25T14:20:42.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.667+0000 7f459970e640 1 -- 172.21.15.63:0/276097219 learned_addr learned my addr 172.21.15.63:0/276097219 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:42.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.673+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245009010 0 0) 0x7f45940b0c70 con 0x7f459406adf0 2024-03-25T14:20:42.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.673+0000 7f458bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:20:42.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.673+0000 7f458bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:42.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.673+0000 7f458bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:42.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.673+0000 7f458bfff640 10 monclient(hunting): my global_id is 4267 2024-03-25T14:20:42.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.673+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580003520 con 0x7f459406adf0 2024-03-25T14:20:42.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2565716261 0 0) 0x7f4580003520 con 0x7f459406adf0 2024-03-25T14:20:42.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 >> v1:172.21.15.63:6790/0 conn(0x7f45940aee10 legacy=0x7f4594140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:20:42.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 >> v1:172.21.15.63:6789/0 conn(0x7f4594140860 legacy=0x7f459414b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:42.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:20:42.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 1 monclient: found mon.noname-c 2024-03-25T14:20:42.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:42.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459413dc40 con 0x7f459406adf0 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 10 monclient: _finish_auth 0 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4590001690 con 0x7f459406adf0 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f458bfff640 10 monclient: dump: 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.686 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:42.687 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:42.687 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:42.687 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:42.687 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:42.687 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:42.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f459b972640 5 monclient: authenticate success, global_id 4267 2024-03-25T14:20:42.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.682+0000 7f459b972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:42.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4590001860 con 0x7f459406adf0 2024-03-25T14:20:42.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f458bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:42.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f458bfff640 1 -- 172.21.15.63:0/276097219 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4590001cb0 con 0x7f459406adf0 2024-03-25T14:20:42.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f459b972640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:42.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f459b972640 4 set_mon_vals no callback set 2024-03-25T14:20:42.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f458bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f458bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.702+0000 7f458bfff640 10 monclient: dump: 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:42.706 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:42.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.703+0000 7f459b972640 10 monclient: shutdown 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.703+0000 7f459b972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.703+0000 7f459b972640 1 -- 172.21.15.63:0/276097219 >> v1:172.21.15.153:6789/0 conn(0x7f459406adf0 legacy=0x7f459413d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.703+0000 7f459b972640 1 -- 172.21.15.63:0/276097219 shutdown_connections 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.703+0000 7f459b972640 1 -- 172.21.15.63:0/276097219 >> 172.21.15.63:0/276097219 conn(0x7f45940b4a80 msgr2=0x7f45940b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.703+0000 7f459b972640 1 -- 172.21.15.63:0/276097219 shutdown_connections 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.703+0000 7f459b972640 1 -- 172.21.15.63:0/276097219 wait complete. 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 10 monclient: build_initial_monmap 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 10 monclient: monmap: 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:42.709 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 1 librados: starting msgr at 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 1 librados: starting objecter 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 1 Processor -- start 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 1 -- start start 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 1 librados: setting wanted keys 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 1 librados: calling monclient init 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.704+0000 7f459b972640 10 monclient: init 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:42.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 10 monclient(hunting): picked mon.noname-c con 0x7f459406adf0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 10 monclient(hunting): picked mon.noname-a con 0x7f45940aee10 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 10 monclient(hunting): picked mon.noname-b con 0x7f45941433a0 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4594070940 con 0x7f45940aee10 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4594070c20 con 0x7f45941433a0 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4594141430 con 0x7f459406adf0 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:42.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.705+0000 7f459b972640 10 monclient(hunting): authenticate will time out at 1405.850236s 2024-03-25T14:20:42.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.711+0000 7f4599f0f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f45941433a0 0x7f45941e77d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:52276/0 (socket says 172.21.15.63:52276) 2024-03-25T14:20:42.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.711+0000 7f4599f0f640 1 -- 172.21.15.63:0/3405942830 learned_addr learned my addr 172.21.15.63:0/3405942830 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:42.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.712+0000 7f4589ffb640 10 client.?.objecter ms_handle_connect 0x7f45941433a0 2024-03-25T14:20:42.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.712+0000 7f4589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:42.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.716+0000 7f4589ffb640 10 client.?.objecter ms_handle_connect 0x7f459406adf0 2024-03-25T14:20:42.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.716+0000 7f4589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:42.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.717+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724345060 0 0) 0x7f4594070c20 con 0x7f45941433a0 2024-03-25T14:20:42.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.717+0000 7f4589ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:42.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.717+0000 7f4589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:42.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.717+0000 7f4589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:42.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.717+0000 7f4589ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T14:20:42.720 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.717+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4578004eb0 con 0x7f45941433a0 2024-03-25T14:20:42.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.721+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101313061 0 0) 0x7f4594141430 con 0x7f459406adf0 2024-03-25T14:20:42.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.721+0000 7f4589ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:42.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.721+0000 7f4589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:42.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.721+0000 7f4589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:42.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.721+0000 7f4589ffb640 10 monclient(hunting): my global_id is 4253 2024-03-25T14:20:42.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.721+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4594070c20 con 0x7f459406adf0 2024-03-25T14:20:42.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.734+0000 7f4589ffb640 10 client.?.objecter ms_handle_connect 0x7f45940aee10 2024-03-25T14:20:42.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.734+0000 7f4589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:42.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1317857040 0 0) 0x7f4578004eb0 con 0x7f45941433a0 2024-03-25T14:20:42.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4594141430 con 0x7f45941433a0 2024-03-25T14:20:42.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f458c001c90 con 0x7f45941433a0 2024-03-25T14:20:42.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:42.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:42.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): dump: 2024-03-25T14:20:42.747 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:42.748 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:42.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:42.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842878923 0 0) 0x7f4594070940 con 0x7f45940aee10 2024-03-25T14:20:42.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:42.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:42.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:42.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 10 monclient(hunting): my global_id is 4254 2024-03-25T14:20:42.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.744+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4578004eb0 con 0x7f45940aee10 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.758+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320972320 0 0) 0x7f4594070c20 con 0x7f459406adf0 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.758+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4594070940 con 0x7f459406adf0 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.758+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4590001b70 con 0x7f459406adf0 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.758+0000 7f4589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.758+0000 7f4589ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.758+0000 7f4589ffb640 10 monclient(hunting): dump: 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.761 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.762 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:42.762 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:42.762 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:42.762 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:42.762 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:42.762 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:42.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.762+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2600431793 0 0) 0x7f4578004eb0 con 0x7f45940aee10 2024-03-25T14:20:42.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.762+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4594070c20 con 0x7f45940aee10 2024-03-25T14:20:42.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.762+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4584001790 con 0x7f45940aee10 2024-03-25T14:20:42.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.762+0000 7f4589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:42.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.762+0000 7f4589ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:20:42.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.762+0000 7f4589ffb640 10 monclient(hunting): dump: 2024-03-25T14:20:42.765 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:42.766 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:42.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3886492462 0 0) 0x7f4594141430 con 0x7f45941433a0 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 >> v1:172.21.15.63:6790/0 conn(0x7f459406adf0 legacy=0x7f4594142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 >> v1:172.21.15.63:6789/0 conn(0x7f45940aee10 legacy=0x7f45941e40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 1 monclient: found mon.b 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594139800 con 0x7f45941433a0 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 10 monclient: _finish_auth 0 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f4589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:42.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f459b972640 5 monclient: authenticate success, global_id 4270 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f459b972640 10 monclient: _renew_subs 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.773+0000 7f459b972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.774+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459406fd60 con 0x7f45941433a0 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.774+0000 7f459b972640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.774+0000 7f459b972640 10 monclient: _renew_subs 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.774+0000 7f459b972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.774+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f459413c970 con 0x7f45941433a0 2024-03-25T14:20:42.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.774+0000 7f459b972640 1 librados: init done 2024-03-25T14:20:42.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.775+0000 7f459b972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:42.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.775+0000 7f459b972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:42.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.775+0000 7f459b972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:42.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:42.775+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459406b1d0 con 0x7f45941433a0 2024-03-25T14:20:43.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.062+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f458c003990 con 0x7f45941433a0 2024-03-25T14:20:43.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.062+0000 7f4589ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:43.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.062+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f458c003d60 con 0x7f45941433a0 2024-03-25T14:20:43.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.062+0000 7f4589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.062+0000 7f4589ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.062+0000 7f4589ffb640 10 monclient: dump: 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:43.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.062+0000 7f458b7fe640 4 set_mon_vals no callback set 2024-03-25T14:20:43.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.063+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f458c003f80 con 0x7f45941433a0 2024-03-25T14:20:43.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.064+0000 7f459970e640 10 monclient: get_auth_request con 0x7f457806e110 auth_method 0 2024-03-25T14:20:43.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.064+0000 7f4589ffb640 10 client.4270.objecter ms_handle_connect 0x7f457806e110 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f458c054b20 con 0x7f45941433a0 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 10 client.4270.objecter ms_dispatch 0x7f4594000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f458c050630 con 0x7f45941433a0 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:43.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.070+0000 7f4589ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:43.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.292+0000 7f459b972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:43.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.292+0000 7f459b972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:43.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.292+0000 7f459b972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:43.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.292+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f459413cc70 con 0x7f45941433a0 2024-03-25T14:20:43.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.345+0000 7f4589ffb640 1 -- 172.21.15.63:0/3405942830 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f458c052560 con 0x7f45941433a0 2024-03-25T14:20:43.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.345+0000 7f4589ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:43.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.345+0000 7f4589ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:43.348 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:43.348 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:43.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.347+0000 7f459b972640 10 librados: watch_flush enter 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.347+0000 7f459b972640 10 librados: watch_flush exit 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.347+0000 7f459b972640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.347+0000 7f459b972640 10 client.4270.objecter successfully canceled tick 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.347+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 >> v1:172.21.15.153:6800/1688255552 conn(0x7f457806e110 legacy=0x7f45780705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 10 monclient: shutdown 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 >> v1:172.21.15.153:6789/0 conn(0x7f45941433a0 legacy=0x7f45941e77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 shutdown_connections 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 >> 172.21.15.63:0/3405942830 conn(0x7f45940b4a80 msgr2=0x7f459413f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 shutdown_connections 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 1 -- 172.21.15.63:0/3405942830 wait complete. 2024-03-25T14:20:43.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:43.348+0000 7f459b972640 1 librados: shutdown 2024-03-25T14:20:43.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:45.922 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:24.436 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:45.986 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:24.500 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:46.039 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:24.500 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:46.102 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:24.563 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:46.136 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:24.650 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:46.197 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:24.711 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:46.228 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:24.742 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:46.340 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:24.801 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:46.442 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:24.956 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:20:50.168 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:50.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.284+0000 7fa3202e2640 10 monclient: get_monmap_and_config 2024-03-25T14:20:50.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.284+0000 7fa3202e2640 10 monclient: build_initial_monmap 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.284+0000 7fa3202e2640 10 monclient: monmap: 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:50.285142+0000 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:50.285142+0000 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:50.287 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.284+0000 7fa3202e2640 1 Processor -- start 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.284+0000 7fa3202e2640 1 -- start start 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.284+0000 7fa3202e2640 10 monclient: init 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa3180a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa31813f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa3181430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa318142520 con 0x7fa31813f710 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3180b28a0 con 0x7fa3181430c0 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3180b2a30 con 0x7fa3180a98f0 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:50.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.285+0000 7fa3202e2640 10 monclient(hunting): authenticate will time out at 1413.430036s 2024-03-25T14:20:50.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.294+0000 7fa31d87d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fa31813f710 0x7fa3181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:39674/0 (socket says 172.21.15.63:39674) 2024-03-25T14:20:50.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.294+0000 7fa31d87d640 1 -- 172.21.15.63:0/2784564584 learned_addr learned my addr 172.21.15.63:0/2784564584 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:50.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.302+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983027690 0 0) 0x7fa3180b28a0 con 0x7fa3181430c0 2024-03-25T14:20:50.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.302+0000 7fa31d07c640 10 monclient(hunting): _negotiate 2024-03-25T14:20:50.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.302+0000 7fa31d07c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:50.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.302+0000 7fa31d07c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:50.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.302+0000 7fa31d07c640 10 monclient(hunting): my global_id is 4273 2024-03-25T14:20:50.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.302+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304003520 con 0x7fa3181430c0 2024-03-25T14:20:50.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.307+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600549979 0 0) 0x7fa318142520 con 0x7fa31813f710 2024-03-25T14:20:50.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.307+0000 7fa31d07c640 10 monclient(hunting): _negotiate 2024-03-25T14:20:50.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.307+0000 7fa31d07c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:50.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.307+0000 7fa31d07c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:50.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.307+0000 7fa31d07c640 10 monclient(hunting): my global_id is 4257 2024-03-25T14:20:50.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.307+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3180b28a0 con 0x7fa31813f710 2024-03-25T14:20:50.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.308+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804059271 0 0) 0x7fa3180b2a30 con 0x7fa3180a98f0 2024-03-25T14:20:50.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.308+0000 7fa31d07c640 10 monclient(hunting): _negotiate 2024-03-25T14:20:50.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.308+0000 7fa31d07c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:50.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.308+0000 7fa31d07c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:50.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.308+0000 7fa31d07c640 10 monclient(hunting): my global_id is 4256 2024-03-25T14:20:50.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.308+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa318142520 con 0x7fa3180a98f0 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1816895082 0 0) 0x7fa318142520 con 0x7fa3180a98f0 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 >> v1:172.21.15.153:6789/0 conn(0x7fa3181430c0 legacy=0x7fa31814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 >> v1:172.21.15.63:6789/0 conn(0x7fa31813f710 legacy=0x7fa3181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 10 monclient: _finish_hunting 0 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 1 monclient: found mon.noname-b 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3180b44c0 con 0x7fa3180a98f0 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 10 monclient: _finish_auth 0 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 10 monclient: _check_auth_tickets 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:50.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa3140015b0 con 0x7fa3180a98f0 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa31d07c640 10 monclient: dump: 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa3202e2640 5 monclient: authenticate success, global_id 4256 2024-03-25T14:20:50.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.315+0000 7fa3202e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa314001710 con 0x7fa3180a98f0 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa31d07c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa31d07c640 1 -- 172.21.15.63:0/2784564584 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa314001b60 con 0x7fa3180a98f0 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa31d07c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa31d07c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa31d07c640 10 monclient: dump: 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:50.333 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa3202e2640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:50.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.330+0000 7fa3202e2640 4 set_mon_vals no callback set 2024-03-25T14:20:50.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.331+0000 7fa3202e2640 10 monclient: shutdown 2024-03-25T14:20:50.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.331+0000 7fa3202e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.331+0000 7fa3202e2640 1 -- 172.21.15.63:0/2784564584 >> v1:172.21.15.63:6790/0 conn(0x7fa3180a98f0 legacy=0x7fa3180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.331+0000 7fa3202e2640 1 -- 172.21.15.63:0/2784564584 shutdown_connections 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.331+0000 7fa3202e2640 1 -- 172.21.15.63:0/2784564584 >> 172.21.15.63:0/2784564584 conn(0x7fa3180b7ea0 msgr2=0x7fa3180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.331+0000 7fa3202e2640 1 -- 172.21.15.63:0/2784564584 shutdown_connections 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.331+0000 7fa3202e2640 1 -- 172.21.15.63:0/2784564584 wait complete. 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.332+0000 7fa3202e2640 10 monclient: build_initial_monmap 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.332+0000 7fa3202e2640 10 monclient: monmap: 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:50.339 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.333+0000 7fa3202e2640 1 librados: starting msgr at 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.333+0000 7fa3202e2640 1 librados: starting objecter 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.333+0000 7fa3202e2640 1 Processor -- start 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.333+0000 7fa3202e2640 1 -- start start 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.333+0000 7fa3202e2640 1 librados: setting wanted keys 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.333+0000 7fa3202e2640 1 librados: calling monclient init 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.333+0000 7fa3202e2640 10 monclient: init 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:20:50.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa3180a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:50.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa31813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:50.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa3181430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:50.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa318138050 con 0x7fa3181430c0 2024-03-25T14:20:50.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa318137d70 con 0x7fa31813f710 2024-03-25T14:20:50.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa318138f20 con 0x7fa3180a98f0 2024-03-25T14:20:50.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:50.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.334+0000 7fa3202e2640 10 monclient(hunting): authenticate will time out at 1413.478885s 2024-03-25T14:20:50.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.341+0000 7fa31e87f640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fa3181430c0 0x7fa3181e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:39718/0 (socket says 172.21.15.63:39718) 2024-03-25T14:20:50.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.341+0000 7fa31e87f640 1 -- 172.21.15.63:0/1765097700 learned_addr learned my addr 172.21.15.63:0/1765097700 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:50.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.347+0000 7fa30effd640 10 client.?.objecter ms_handle_connect 0x7fa3181430c0 2024-03-25T14:20:50.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.347+0000 7fa30effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:50.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.351+0000 7fa30effd640 10 client.?.objecter ms_handle_connect 0x7fa3180a98f0 2024-03-25T14:20:50.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.351+0000 7fa30effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:50.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.359+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681821571 0 0) 0x7fa318138050 con 0x7fa3181430c0 2024-03-25T14:20:50.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.359+0000 7fa30effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:50.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.359+0000 7fa30effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:50.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.359+0000 7fa30effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:50.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.359+0000 7fa30effd640 10 monclient(hunting): my global_id is 4260 2024-03-25T14:20:50.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.359+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2fc004eb0 con 0x7fa3181430c0 2024-03-25T14:20:50.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.361+0000 7fa30effd640 10 client.?.objecter ms_handle_connect 0x7fa31813f710 2024-03-25T14:20:50.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.361+0000 7fa30effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:50.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.367+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098406530 0 0) 0x7fa318138f20 con 0x7fa3180a98f0 2024-03-25T14:20:50.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.367+0000 7fa30effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:50.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.367+0000 7fa30effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:50.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.367+0000 7fa30effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:50.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.367+0000 7fa30effd640 10 monclient(hunting): my global_id is 4259 2024-03-25T14:20:50.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.367+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa318138050 con 0x7fa3180a98f0 2024-03-25T14:20:50.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.376+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82917578 0 0) 0x7fa318137d70 con 0x7fa31813f710 2024-03-25T14:20:50.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.376+0000 7fa30effd640 10 monclient(hunting): _negotiate 2024-03-25T14:20:50.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.376+0000 7fa30effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:50.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.376+0000 7fa30effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:50.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.376+0000 7fa30effd640 10 monclient(hunting): my global_id is 4276 2024-03-25T14:20:50.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.376+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa318138f20 con 0x7fa31813f710 2024-03-25T14:20:50.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.385+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3823756704 0 0) 0x7fa318138f20 con 0x7fa31813f710 2024-03-25T14:20:50.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.385+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa318137d70 con 0x7fa31813f710 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2633828658 0 0) 0x7fa318138050 con 0x7fa3180a98f0 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa318138f20 con 0x7fa3180a98f0 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa314001a20 con 0x7fa3180a98f0 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 10 monclient(hunting): dump: 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:50.391 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1285935565 0 0) 0x7fa2fc004eb0 con 0x7fa3181430c0 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa318138050 con 0x7fa3181430c0 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa310001c30 con 0x7fa3181430c0 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:20:50.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.388+0000 7fa30effd640 10 monclient(hunting): dump: 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:50.393 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa308001930 con 0x7fa31813f710 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 10 monclient(hunting): dump: 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:50.401 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 497546302 0 0) 0x7fa318137d70 con 0x7fa31813f710 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 >> v1:172.21.15.63:6790/0 conn(0x7fa3180a98f0 legacy=0x7fa3181db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 >> v1:172.21.15.63:6789/0 conn(0x7fa3181430c0 legacy=0x7fa3181e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 10 monclient: _finish_hunting 0 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 1 monclient: found mon.b 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3181379b0 con 0x7fa31813f710 2024-03-25T14:20:50.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 10 monclient: _finish_auth 0 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 10 monclient: _check_auth_tickets 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa30effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 5 monclient: authenticate success, global_id 4276 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 10 monclient: _renew_subs 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa318138960 con 0x7fa31813f710 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 10 monclient: _renew_subs 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3181e32d0 con 0x7fa31813f710 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.396+0000 7fa3202e2640 1 librados: init done 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.397+0000 7fa3202e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.398+0000 7fa3202e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.398+0000 7fa3202e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:50.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.398+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa318137ba0 con 0x7fa31813f710 2024-03-25T14:20:50.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.671+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3080047e0 con 0x7fa31813f710 2024-03-25T14:20:50.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.671+0000 7fa30effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:50.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.671+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa308004af0 con 0x7fa31813f710 2024-03-25T14:20:50.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.671+0000 7fa30effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:50.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.671+0000 7fa30effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:50.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.671+0000 7fa30effd640 10 monclient: dump: 2024-03-25T14:20:50.674 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:50.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.671+0000 7fa31c87b640 4 set_mon_vals no callback set 2024-03-25T14:20:50.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.672+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fa30801aac0 con 0x7fa31813f710 2024-03-25T14:20:50.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.673+0000 7fa31e07e640 10 monclient: get_auth_request con 0x7fa2fc06e0d0 auth_method 0 2024-03-25T14:20:50.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.673+0000 7fa30effd640 10 client.4276.objecter ms_handle_connect 0x7fa2fc06e0d0 2024-03-25T14:20:50.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.677+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fa308054750 con 0x7fa31813f710 2024-03-25T14:20:50.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.677+0000 7fa30effd640 10 client.4276.objecter ms_dispatch 0x7fa318000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:50.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.677+0000 7fa30effd640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:50.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.677+0000 7fa30effd640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:50.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.677+0000 7fa30effd640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T14:20:50.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.678+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa308050090 con 0x7fa31813f710 2024-03-25T14:20:50.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.678+0000 7fa30effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:50.681 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.678+0000 7fa30effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:50.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.924+0000 7fa3202e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:50.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.924+0000 7fa3202e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:50.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.924+0000 7fa3202e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:50.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.924+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3180627b0 con 0x7fa31813f710 2024-03-25T14:20:50.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.956+0000 7fa30effd640 1 -- 172.21.15.63:0/1765097700 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fa308052190 con 0x7fa31813f710 2024-03-25T14:20:50.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.956+0000 7fa30effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:50.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.956+0000 7fa30effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:50.959 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:50.959 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:50.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 10 librados: watch_flush enter 2024-03-25T14:20:50.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 10 librados: watch_flush exit 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 10 client.4276.objecter successfully canceled tick 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 >> v1:172.21.15.153:6800/1688255552 conn(0x7fa2fc06e0d0 legacy=0x7fa2fc070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 10 monclient: shutdown 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 >> v1:172.21.15.153:6789/0 conn(0x7fa31813f710 legacy=0x7fa318138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 shutdown_connections 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 >> 172.21.15.63:0/1765097700 conn(0x7fa3180b7ea0 msgr2=0x7fa318074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 shutdown_connections 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.958+0000 7fa3202e2640 1 -- 172.21.15.63:0/1765097700 wait complete. 2024-03-25T14:20:50.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:50.959+0000 7fa3202e2640 1 librados: shutdown 2024-03-25T14:20:50.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:20:57.782 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.897+0000 7fe09d435640 10 monclient: get_monmap_and_config 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.897+0000 7fe09d435640 10 monclient: build_initial_monmap 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.897+0000 7fe09d435640 10 monclient: monmap: 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:20:57.898613+0000 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:20:57.898613+0000 2024-03-25T14:20:57.900 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:57.901 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:57.901 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:57.901 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:20:57.901 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:20:57.901 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:57.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 1 Processor -- start 2024-03-25T14:20:57.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 1 -- start start 2024-03-25T14:20:57.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient: init 2024-03-25T14:20:57.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:57.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:20:57.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient(hunting): picked mon.noname-c con 0x7fe0980b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:57.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient(hunting): picked mon.noname-b con 0x7fe0980b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:57.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient(hunting): picked mon.noname-a con 0x7fe09814be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:57.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0980af940 con 0x7fe09814be90 2024-03-25T14:20:57.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0980b0a70 con 0x7fe0980b4d60 2024-03-25T14:20:57.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0980b0d50 con 0x7fe0980b1730 2024-03-25T14:20:57.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:57.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.898+0000 7fe09d435640 10 monclient(hunting): authenticate will time out at 1421.043440s 2024-03-25T14:20:57.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.903+0000 7fe0967fc640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe0980b1730 0x7fe0980b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:60934/0 (socket says 172.21.15.63:60934) 2024-03-25T14:20:57.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.903+0000 7fe0967fc640 1 -- 172.21.15.63:0/3866091988 learned_addr learned my addr 172.21.15.63:0/3866091988 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:57.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.913+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871940315 0 0) 0x7fe0980b0d50 con 0x7fe0980b1730 2024-03-25T14:20:57.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.913+0000 7fe095ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:57.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.913+0000 7fe095ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:57.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.913+0000 7fe095ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:57.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.913+0000 7fe095ffb640 10 monclient(hunting): my global_id is 4262 2024-03-25T14:20:57.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.913+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe080003520 con 0x7fe0980b1730 2024-03-25T14:20:57.925 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.923+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592468364 0 0) 0x7fe0980af940 con 0x7fe09814be90 2024-03-25T14:20:57.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.923+0000 7fe095ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:57.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.923+0000 7fe095ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:57.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.923+0000 7fe095ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:57.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.923+0000 7fe095ffb640 10 monclient(hunting): my global_id is 4263 2024-03-25T14:20:57.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.923+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0980b0d50 con 0x7fe09814be90 2024-03-25T14:20:57.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.924+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160821594 0 0) 0x7fe0980b0a70 con 0x7fe0980b4d60 2024-03-25T14:20:57.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.924+0000 7fe095ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:20:57.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.924+0000 7fe095ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:57.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.924+0000 7fe095ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:57.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.924+0000 7fe095ffb640 10 monclient(hunting): my global_id is 4279 2024-03-25T14:20:57.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.924+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0980af940 con 0x7fe0980b4d60 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756359555 0 0) 0x7fe080003520 con 0x7fe0980b1730 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 >> v1:172.21.15.153:6789/0 conn(0x7fe0980b4d60 legacy=0x7fe09813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 >> v1:172.21.15.63:6789/0 conn(0x7fe09814be90 legacy=0x7fe09814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 1 monclient: found mon.noname-b 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0980b1030 con 0x7fe0980b1730 2024-03-25T14:20:57.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 10 monclient: _finish_auth 0 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe08c001690 con 0x7fe0980b1730 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe09d435640 5 monclient: authenticate success, global_id 4262 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe09d435640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe095ffb640 10 monclient: dump: 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:57.933 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:57.934 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:57.934 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:57.934 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:57.934 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:57.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.929+0000 7fe09d435640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:20:57.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.933+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe08c0017f0 con 0x7fe0980b1730 2024-03-25T14:20:57.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe095ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:57.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe095ffb640 1 -- 172.21.15.63:0/3866091988 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe08c001c00 con 0x7fe0980b1730 2024-03-25T14:20:57.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe095ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe095ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe095ffb640 10 monclient: dump: 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe09d435640 10 monclient: get_monmap_and_config success 2024-03-25T14:20:57.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe09d435640 4 set_mon_vals no callback set 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe09d435640 10 monclient: shutdown 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe09d435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.934+0000 7fe09d435640 1 -- 172.21.15.63:0/3866091988 >> v1:172.21.15.63:6790/0 conn(0x7fe0980b1730 legacy=0x7fe0980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.935+0000 7fe09d435640 1 -- 172.21.15.63:0/3866091988 shutdown_connections 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.935+0000 7fe09d435640 1 -- 172.21.15.63:0/3866091988 >> 172.21.15.63:0/3866091988 conn(0x7fe0980617a0 msgr2=0x7fe098061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.935+0000 7fe09d435640 1 -- 172.21.15.63:0/3866091988 shutdown_connections 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.935+0000 7fe09d435640 1 -- 172.21.15.63:0/3866091988 wait complete. 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.936+0000 7fe09d435640 10 monclient: build_initial_monmap 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.936+0000 7fe09d435640 10 monclient: monmap: 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:20:57.939 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.936+0000 7fe09d435640 1 librados: starting msgr at 2024-03-25T14:20:57.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.936+0000 7fe09d435640 1 librados: starting objecter 2024-03-25T14:20:57.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.936+0000 7fe09d435640 1 Processor -- start 2024-03-25T14:20:57.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 1 -- start start 2024-03-25T14:20:57.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 1 librados: setting wanted keys 2024-03-25T14:20:57.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 1 librados: calling monclient init 2024-03-25T14:20:57.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient: init 2024-03-25T14:20:57.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient: _reopen_session rank -1 2024-03-25T14:20:57.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:20:57.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient(hunting): picked mon.noname-b con 0x7fe0980b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:20:57.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient(hunting): picked mon.noname-c con 0x7fe0980b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:20:57.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient(hunting): picked mon.noname-a con 0x7fe09814be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:20:57.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe098137d90 con 0x7fe09814be90 2024-03-25T14:20:57.943 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe098142110 con 0x7fe0980b1730 2024-03-25T14:20:57.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe098140b20 con 0x7fe0980b4d60 2024-03-25T14:20:57.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient(hunting): _renew_subs 2024-03-25T14:20:57.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.937+0000 7fe09d435640 10 monclient(hunting): authenticate will time out at 1421.082380s 2024-03-25T14:20:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.947+0000 7fe0967fc640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe0980b4d60 0x7fe098133fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:60950/0 (socket says 172.21.15.63:60950) 2024-03-25T14:20:57.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.947+0000 7fe0967fc640 1 -- 172.21.15.63:0/2493470474 learned_addr learned my addr 172.21.15.63:0/2493470474 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:20:57.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.948+0000 7fe07f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0980b1730 2024-03-25T14:20:57.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.948+0000 7fe07f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:57.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.948+0000 7fe07f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0980b4d60 2024-03-25T14:20:57.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.948+0000 7fe07f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:57.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.954+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93249698 0 0) 0x7fe098142110 con 0x7fe0980b1730 2024-03-25T14:20:57.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.954+0000 7fe07f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:20:57.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.954+0000 7fe07f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:57.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.954+0000 7fe07f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:57.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.954+0000 7fe07f7fe640 10 monclient(hunting): my global_id is 4282 2024-03-25T14:20:57.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.954+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe084004eb0 con 0x7fe0980b1730 2024-03-25T14:20:57.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.955+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271941531 0 0) 0x7fe098140b20 con 0x7fe0980b4d60 2024-03-25T14:20:57.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.955+0000 7fe07f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:20:57.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.955+0000 7fe07f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:57.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.955+0000 7fe07f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:57.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.955+0000 7fe07f7fe640 10 monclient(hunting): my global_id is 4265 2024-03-25T14:20:57.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.955+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe098142110 con 0x7fe0980b4d60 2024-03-25T14:20:57.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.959+0000 7fe07f7fe640 10 client.?.objecter ms_handle_connect 0x7fe09814be90 2024-03-25T14:20:57.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.959+0000 7fe07f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.969+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985487203 0 0) 0x7fe084004eb0 con 0x7fe0980b1730 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.969+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe098140b20 con 0x7fe0980b1730 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.969+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe0880018f0 con 0x7fe0980b1730 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.969+0000 7fe07f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.969+0000 7fe07f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.969+0000 7fe07f7fe640 10 monclient(hunting): dump: 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:57.972 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:57.973 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:57.973 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:57.973 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:57.973 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:57.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.969+0000 7fe07f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:20:57.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571890195 0 0) 0x7fe098137d90 con 0x7fe09814be90 2024-03-25T14:20:57.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:20:57.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:20:57.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:20:57.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T14:20:57.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe084004eb0 con 0x7fe09814be90 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 924889174 0 0) 0x7fe098140b20 con 0x7fe0980b1730 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 >> v1:172.21.15.63:6790/0 conn(0x7fe0980b4d60 legacy=0x7fe098133fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 >> v1:172.21.15.63:6789/0 conn(0x7fe09814be90 legacy=0x7fe0981e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 1 monclient: found mon.b 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:57.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe098140cb0 con 0x7fe0980b1730 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe07f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.982+0000 7fe09d435640 5 monclient: authenticate success, global_id 4282 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 10 monclient: _renew_subs 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe098140ef0 con 0x7fe0980b1730 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 10 monclient: _renew_subs 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe098142710 con 0x7fe0980b1730 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.983+0000 7fe09d435640 1 librados: init done 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.984+0000 7fe09d435640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.984+0000 7fe09d435640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:57.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.984+0000 7fe09d435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:57.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:57.984+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0980b2270 con 0x7fe0980b1730 2024-03-25T14:20:58.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.266+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe088003610 con 0x7fe0980b1730 2024-03-25T14:20:58.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.266+0000 7fe07f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:20:58.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.266+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe0880039a0 con 0x7fe0980b1730 2024-03-25T14:20:58.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.266+0000 7fe07f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:20:58.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.266+0000 7fe07f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:20:58.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.266+0000 7fe07f7fe640 10 monclient: dump: 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:20:58.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.266+0000 7fe0957fa640 4 set_mon_vals no callback set 2024-03-25T14:20:58.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.267+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fe088003bc0 con 0x7fe0980b1730 2024-03-25T14:20:58.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.268+0000 7fe0967fc640 10 monclient: get_auth_request con 0x7fe08406de00 auth_method 0 2024-03-25T14:20:58.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.268+0000 7fe07f7fe640 10 client.4282.objecter ms_handle_connect 0x7fe08406de00 2024-03-25T14:20:58.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.271+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fe088053600 con 0x7fe0980b1730 2024-03-25T14:20:58.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.271+0000 7fe07f7fe640 10 client.4282.objecter ms_dispatch 0x7fe098000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:20:58.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.271+0000 7fe07f7fe640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:20:58.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.271+0000 7fe07f7fe640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:20:58.275 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.271+0000 7fe07f7fe640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T14:20:58.275 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.272+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe08804f110 con 0x7fe0980b1730 2024-03-25T14:20:58.275 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.272+0000 7fe07f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:20:58.275 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.272+0000 7fe07f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:20:58.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.526+0000 7fe09d435640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:58.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.526+0000 7fe09d435640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:58.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.526+0000 7fe09d435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:20:58.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.526+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0980627b0 con 0x7fe0980b1730 2024-03-25T14:20:58.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.559+0000 7fe07f7fe640 1 -- 172.21.15.63:0/2493470474 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fe088051040 con 0x7fe0980b1730 2024-03-25T14:20:58.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.559+0000 7fe07f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:20:58.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.559+0000 7fe07f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:20:58.563 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:20:58.563 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:20:58.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 10 librados: watch_flush enter 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 10 librados: watch_flush exit 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 10 client.4282.objecter successfully canceled tick 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 >> v1:172.21.15.153:6800/1688255552 conn(0x7fe08406de00 legacy=0x7fe0840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 10 monclient: shutdown 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 >> v1:172.21.15.153:6789/0 conn(0x7fe0980b1730 legacy=0x7fe098140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 shutdown_connections 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 >> 172.21.15.63:0/2493470474 conn(0x7fe0980617a0 msgr2=0x7fe09813b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.562+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 shutdown_connections 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.563+0000 7fe09d435640 1 -- 172.21.15.63:0/2493470474 wait complete. 2024-03-25T14:20:58.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:20:58.563+0000 7fe09d435640 1 librados: shutdown 2024-03-25T14:20:58.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:05.385 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:05.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.487+0000 7f85f6f25640 10 monclient: get_monmap_and_config 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.487+0000 7f85f6f25640 10 monclient: build_initial_monmap 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.487+0000 7f85f6f25640 10 monclient: monmap: 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:05.488930+0000 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:05.488930+0000 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:05.491 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:05.492 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:05.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.488+0000 7f85f6f25640 1 Processor -- start 2024-03-25T14:21:05.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.488+0000 7f85f6f25640 1 -- start start 2024-03-25T14:21:05.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.488+0000 7f85f6f25640 10 monclient: init 2024-03-25T14:21:05.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 10 monclient(hunting): picked mon.noname-a con 0x7f85f00a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 10 monclient(hunting): picked mon.noname-b con 0x7f85f013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 10 monclient(hunting): picked mon.noname-c con 0x7f85f01430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f0142520 con 0x7f85f00a98f0 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00b28a0 con 0x7f85f01430c0 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00b2a30 con 0x7f85f013f710 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:05.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.489+0000 7f85f6f25640 10 monclient(hunting): authenticate will time out at 1428.633932s 2024-03-25T14:21:05.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.525+0000 7f85f54c2640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f85f01430c0 0x7f85f014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:48470/0 (socket says 172.21.15.63:48470) 2024-03-25T14:21:05.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.525+0000 7f85f54c2640 1 -- 172.21.15.63:0/106091490 learned_addr learned my addr 172.21.15.63:0/106091490 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:05.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.557+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060275853 0 0) 0x7f85f00b28a0 con 0x7f85f01430c0 2024-03-25T14:21:05.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.557+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:05.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.557+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:05.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.557+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:05.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.557+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 4285 2024-03-25T14:21:05.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.557+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e0003520 con 0x7f85f01430c0 2024-03-25T14:21:05.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.565+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091655176 0 0) 0x7f85f0142520 con 0x7f85f00a98f0 2024-03-25T14:21:05.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.565+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:05.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.565+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:05.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.565+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:05.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.565+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T14:21:05.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.565+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f00b28a0 con 0x7f85f00a98f0 2024-03-25T14:21:05.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.571+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531294731 0 0) 0x7f85f00b2a30 con 0x7f85f013f710 2024-03-25T14:21:05.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.571+0000 7f85ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:05.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.571+0000 7f85ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:05.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.571+0000 7f85ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:05.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.571+0000 7f85ef7fe640 10 monclient(hunting): my global_id is 4268 2024-03-25T14:21:05.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.571+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0142520 con 0x7f85f013f710 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.574+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 621081332 0 0) 0x7f85e0003520 con 0x7f85f01430c0 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 >> v1:172.21.15.63:6790/0 conn(0x7f85f013f710 legacy=0x7f85f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 >> v1:172.21.15.63:6789/0 conn(0x7f85f00a98f0 legacy=0x7f85f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 1 monclient: found mon.noname-c 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00b44c0 con 0x7f85f01430c0 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 10 monclient: _finish_auth 0 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:21:05.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85f6f25640 5 monclient: authenticate success, global_id 4285 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85f6f25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f85e8001a10 con 0x7f85f01430c0 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85ef7fe640 10 monclient: dump: 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:05.579 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:05.580 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:05.580 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:05.580 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:05.580 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:05.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.575+0000 7f85f6f25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e8001b70 con 0x7f85f01430c0 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85ef7fe640 1 -- 172.21.15.63:0/106091490 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f85e8001f80 con 0x7f85f01430c0 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85ef7fe640 10 monclient: dump: 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.592 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85f6f25640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:05.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.589+0000 7f85f6f25640 4 set_mon_vals no callback set 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.590+0000 7f85f6f25640 10 monclient: shutdown 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.590+0000 7f85f6f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.590+0000 7f85f6f25640 1 -- 172.21.15.63:0/106091490 >> v1:172.21.15.153:6789/0 conn(0x7f85f01430c0 legacy=0x7f85f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.590+0000 7f85f6f25640 1 -- 172.21.15.63:0/106091490 shutdown_connections 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.590+0000 7f85f6f25640 1 -- 172.21.15.63:0/106091490 >> 172.21.15.63:0/106091490 conn(0x7f85f00b7ea0 msgr2=0x7f85f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.590+0000 7f85f6f25640 1 -- 172.21.15.63:0/106091490 shutdown_connections 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.590+0000 7f85f6f25640 1 -- 172.21.15.63:0/106091490 wait complete. 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 10 monclient: build_initial_monmap 2024-03-25T14:21:05.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 10 monclient: monmap: 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 1 librados: starting msgr at 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 1 librados: starting objecter 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 1 Processor -- start 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 1 -- start start 2024-03-25T14:21:05.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 1 librados: setting wanted keys 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 1 librados: calling monclient init 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.591+0000 7f85f6f25640 10 monclient: init 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 10 monclient(hunting): picked mon.noname-a con 0x7f85f00a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 10 monclient(hunting): picked mon.noname-c con 0x7f85f013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 10 monclient(hunting): picked mon.noname-b con 0x7f85f01430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f01369d0 con 0x7f85f00a98f0 2024-03-25T14:21:05.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f0133a60 con 0x7f85f01430c0 2024-03-25T14:21:05.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00ab9e0 con 0x7f85f013f710 2024-03-25T14:21:05.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:05.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.592+0000 7f85f6f25640 10 monclient(hunting): authenticate will time out at 1428.737246s 2024-03-25T14:21:05.609 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.606+0000 7f85f4cc1640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f85f00a98f0 0x7f85f0132c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:55260/0 (socket says 172.21.15.63:55260) 2024-03-25T14:21:05.609 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.606+0000 7f85f4cc1640 1 -- 172.21.15.63:0/3566394723 learned_addr learned my addr 172.21.15.63:0/3566394723 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:05.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.609+0000 7f85ed7fa640 10 client.?.objecter ms_handle_connect 0x7f85f00a98f0 2024-03-25T14:21:05.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.609+0000 7f85ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:05.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.611+0000 7f85ed7fa640 10 client.?.objecter ms_handle_connect 0x7f85f013f710 2024-03-25T14:21:05.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.611+0000 7f85ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:05.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.611+0000 7f85ed7fa640 10 client.?.objecter ms_handle_connect 0x7f85f01430c0 2024-03-25T14:21:05.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.611+0000 7f85ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:05.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.617+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029652599 0 0) 0x7f85f00ab9e0 con 0x7f85f013f710 2024-03-25T14:21:05.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.617+0000 7f85ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:05.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.617+0000 7f85ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:05.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.617+0000 7f85ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:05.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.617+0000 7f85ed7fa640 10 monclient(hunting): my global_id is 4271 2024-03-25T14:21:05.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.618+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85f013f710 2024-03-25T14:21:05.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.621+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450466678 0 0) 0x7f85f01369d0 con 0x7f85f00a98f0 2024-03-25T14:21:05.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.621+0000 7f85ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:05.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.621+0000 7f85ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:05.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.621+0000 7f85ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:05.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.621+0000 7f85ed7fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T14:21:05.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.621+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f00ab9e0 con 0x7f85f00a98f0 2024-03-25T14:21:05.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.624+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796967643 0 0) 0x7f85f0133a60 con 0x7f85f01430c0 2024-03-25T14:21:05.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.624+0000 7f85ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:05.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.624+0000 7f85ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:05.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.624+0000 7f85ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:05.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.624+0000 7f85ed7fa640 10 monclient(hunting): my global_id is 4288 2024-03-25T14:21:05.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.624+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f01369d0 con 0x7f85f01430c0 2024-03-25T14:21:05.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.631+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3676735457 0 0) 0x7f85d4004eb0 con 0x7f85f013f710 2024-03-25T14:21:05.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.631+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85f0133a60 con 0x7f85f013f710 2024-03-25T14:21:05.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.631+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f85dc0018f0 con 0x7f85f013f710 2024-03-25T14:21:05.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.631+0000 7f85ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.631+0000 7f85ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.631+0000 7f85ed7fa640 10 monclient(hunting): dump: 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:05.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.631+0000 7f85ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.641+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3543362825 0 0) 0x7f85f01369d0 con 0x7f85f01430c0 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.642+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85f01430c0 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.642+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f85e8001e80 con 0x7f85f01430c0 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.642+0000 7f85ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.642+0000 7f85ed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.642+0000 7f85ed7fa640 10 monclient(hunting): dump: 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:05.645 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:05.646 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:05.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1921436305 0 0) 0x7f85f0133a60 con 0x7f85f013f710 2024-03-25T14:21:05.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 >> v1:172.21.15.153:6789/0 conn(0x7f85f01430c0 legacy=0x7f85f0133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 >> v1:172.21.15.63:6789/0 conn(0x7f85f00a98f0 legacy=0x7f85f0132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 1 monclient: found mon.c 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00abcc0 con 0x7f85f013f710 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 10 monclient: _finish_auth 0 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.653+0000 7f85ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 5 monclient: authenticate success, global_id 4271 2024-03-25T14:21:05.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 10 monclient: _renew_subs 2024-03-25T14:21:05.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:05.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f00abfa0 con 0x7f85f013f710 2024-03-25T14:21:05.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:05.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 10 monclient: _renew_subs 2024-03-25T14:21:05.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:05.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f0137e40 con 0x7f85f013f710 2024-03-25T14:21:05.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.654+0000 7f85f6f25640 1 librados: init done 2024-03-25T14:21:05.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.655+0000 7f85f6f25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:05.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.655+0000 7f85f6f25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:05.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.655+0000 7f85f6f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:05.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.655+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f00627b0 con 0x7f85f013f710 2024-03-25T14:21:05.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.990+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85dc003610 con 0x7f85f013f710 2024-03-25T14:21:05.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.990+0000 7f85ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.990+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f85dc003a20 con 0x7f85f013f710 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.990+0000 7f85ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.990+0000 7f85ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.990+0000 7f85ed7fa640 10 monclient: dump: 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:05.994 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:05.995 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:05.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.990+0000 7f85eeffd640 4 set_mon_vals no callback set 2024-03-25T14:21:05.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.991+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f85dc003c40 con 0x7f85f013f710 2024-03-25T14:21:05.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.992+0000 7f85f4cc1640 10 monclient: get_auth_request con 0x7f85d406df70 auth_method 0 2024-03-25T14:21:05.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.992+0000 7f85ed7fa640 10 client.4271.objecter ms_handle_connect 0x7f85d406df70 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f85dc053650 con 0x7f85f013f710 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 10 client.4271.objecter ms_dispatch 0x7f85f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85dc04f160 con 0x7f85f013f710 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:06.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:05.998+0000 7f85ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:06.027 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:44.541 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.074 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:44.534 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.078 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:44.539 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.102 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:44.615 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.118 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:44.579 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.149 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:01:44.663 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.151 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:44.665 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.156 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:44.670 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.213 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:01:44.727 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:06.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.217+0000 7f85f6f25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:06.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.217+0000 7f85f6f25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:06.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.217+0000 7f85f6f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:06.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.217+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85f0139000 con 0x7f85f013f710 2024-03-25T14:21:06.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.289+0000 7f85ed7fa640 1 -- 172.21.15.63:0/3566394723 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f85dc051090 con 0x7f85f013f710 2024-03-25T14:21:06.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.290+0000 7f85ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:06.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.290+0000 7f85ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:06.293 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:06.293 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:06.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 10 librados: watch_flush enter 2024-03-25T14:21:06.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 10 librados: watch_flush exit 2024-03-25T14:21:06.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-25T14:21:06.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 10 client.4271.objecter successfully canceled tick 2024-03-25T14:21:06.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 >> v1:172.21.15.153:6800/1688255552 conn(0x7f85d406df70 legacy=0x7f85d4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:06.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 10 monclient: shutdown 2024-03-25T14:21:06.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:06.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 >> v1:172.21.15.63:6790/0 conn(0x7f85f013f710 legacy=0x7f85f01362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:06.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 shutdown_connections 2024-03-25T14:21:06.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 >> 172.21.15.63:0/3566394723 conn(0x7f85f00b7ea0 msgr2=0x7f85f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:06.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 shutdown_connections 2024-03-25T14:21:06.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 1 -- 172.21.15.63:0/3566394723 wait complete. 2024-03-25T14:21:06.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:06.293+0000 7f85f6f25640 1 librados: shutdown 2024-03-25T14:21:06.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:13.115 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:13.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.227+0000 7fd0302dd640 10 monclient: get_monmap_and_config 2024-03-25T14:21:13.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.227+0000 7fd0302dd640 10 monclient: build_initial_monmap 2024-03-25T14:21:13.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.227+0000 7fd0302dd640 10 monclient: monmap: 2024-03-25T14:21:13.230 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:13.230 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:13.228465+0000 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:13.228465+0000 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:13.231 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:13.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.227+0000 7fd0302dd640 1 Processor -- start 2024-03-25T14:21:13.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.228+0000 7fd0302dd640 1 -- start start 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.228+0000 7fd0302dd640 10 monclient: init 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.228+0000 7fd0302dd640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.228+0000 7fd0302dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.228+0000 7fd0302dd640 10 monclient(hunting): picked mon.noname-a con 0x7fd0280a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd0302dd640 10 monclient(hunting): picked mon.noname-b con 0x7fd02813f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd0302dd640 10 monclient(hunting): picked mon.noname-c con 0x7fd0281430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd0302dd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd028142520 con 0x7fd0280a98f0 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd0302dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0280b28a0 con 0x7fd0281430c0 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd0302dd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0280b2a30 con 0x7fd02813f710 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd0302dd640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:13.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd0302dd640 10 monclient(hunting): authenticate will time out at 1436.373671s 2024-03-25T14:21:13.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd02e87a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd0281430c0 0x7fd02814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36464/0 (socket says 172.21.15.63:36464) 2024-03-25T14:21:13.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.229+0000 7fd02e87a640 1 -- 172.21.15.63:0/903704436 learned_addr learned my addr 172.21.15.63:0/903704436 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:13.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.323+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154864668 0 0) 0x7fd028142520 con 0x7fd0280a98f0 2024-03-25T14:21:13.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.323+0000 7fd02d077640 10 monclient(hunting): _negotiate 2024-03-25T14:21:13.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.323+0000 7fd02d077640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:13.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.323+0000 7fd02d077640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:13.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.323+0000 7fd02d077640 10 monclient(hunting): my global_id is 4275 2024-03-25T14:21:13.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.323+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd014003520 con 0x7fd0280a98f0 2024-03-25T14:21:13.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544900591 0 0) 0x7fd014003520 con 0x7fd0280a98f0 2024-03-25T14:21:13.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 >> v1:172.21.15.63:6790/0 conn(0x7fd02813f710 legacy=0x7fd0281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:13.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 >> v1:172.21.15.153:6789/0 conn(0x7fd0281430c0 legacy=0x7fd02814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:13.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 10 monclient: _finish_hunting 0 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 1 monclient: found mon.noname-a 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0280b44c0 con 0x7fd0280a98f0 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 10 monclient: _finish_auth 0 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 10 monclient: _check_auth_tickets 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd02d077640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd0302dd640 5 monclient: authenticate success, global_id 4275 2024-03-25T14:21:13.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.329+0000 7fd0302dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:13.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.332+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd024001690 con 0x7fd0280a98f0 2024-03-25T14:21:13.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.332+0000 7fd02d077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:13.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.332+0000 7fd02d077640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:13.335 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.332+0000 7fd02d077640 10 monclient: dump: 2024-03-25T14:21:13.335 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:13.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.332+0000 7fd0302dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd024001920 con 0x7fd0280a98f0 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd02d077640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd02d077640 1 -- 172.21.15.63:0/903704436 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd024001c30 con 0x7fd0280a98f0 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd02d077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd02d077640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd02d077640 10 monclient: dump: 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.342 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd0302dd640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.338+0000 7fd0302dd640 4 set_mon_vals no callback set 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.339+0000 7fd0302dd640 10 monclient: shutdown 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.339+0000 7fd0302dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.339+0000 7fd0302dd640 1 -- 172.21.15.63:0/903704436 >> v1:172.21.15.63:6789/0 conn(0x7fd0280a98f0 legacy=0x7fd0280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.339+0000 7fd0302dd640 1 -- 172.21.15.63:0/903704436 shutdown_connections 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.339+0000 7fd0302dd640 1 -- 172.21.15.63:0/903704436 >> 172.21.15.63:0/903704436 conn(0x7fd0280b7ea0 msgr2=0x7fd0280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.339+0000 7fd0302dd640 1 -- 172.21.15.63:0/903704436 shutdown_connections 2024-03-25T14:21:13.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.339+0000 7fd0302dd640 1 -- 172.21.15.63:0/903704436 wait complete. 2024-03-25T14:21:13.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 10 monclient: build_initial_monmap 2024-03-25T14:21:13.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 10 monclient: monmap: 2024-03-25T14:21:13.345 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:13.345 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:13.345 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 1 librados: starting msgr at 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 1 librados: starting objecter 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 1 Processor -- start 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 1 -- start start 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 1 librados: setting wanted keys 2024-03-25T14:21:13.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 1 librados: calling monclient init 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.340+0000 7fd0302dd640 10 monclient: init 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 10 monclient(hunting): picked mon.noname-a con 0x7fd0280a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 10 monclient(hunting): picked mon.noname-c con 0x7fd02813f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 10 monclient(hunting): picked mon.noname-b con 0x7fd0281430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0281339d0 con 0x7fd0280a98f0 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd028137800 con 0x7fd0281430c0 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd028135710 con 0x7fd02813f710 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:13.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.341+0000 7fd0302dd640 10 monclient(hunting): authenticate will time out at 1436.486359s 2024-03-25T14:21:13.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.350+0000 7fd02e079640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd0280a98f0 0x7fd028135000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52602/0 (socket says 172.21.15.63:52602) 2024-03-25T14:21:13.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.350+0000 7fd02e079640 1 -- 172.21.15.63:0/475028203 learned_addr learned my addr 172.21.15.63:0/475028203 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:13.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.351+0000 7fd01effd640 10 client.?.objecter ms_handle_connect 0x7fd0280a98f0 2024-03-25T14:21:13.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.351+0000 7fd01effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:13.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.360+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887493795 0 0) 0x7fd0281339d0 con 0x7fd0280a98f0 2024-03-25T14:21:13.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.360+0000 7fd01effd640 10 monclient(hunting): _negotiate 2024-03-25T14:21:13.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.360+0000 7fd01effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:13.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.360+0000 7fd01effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:13.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.360+0000 7fd01effd640 10 monclient(hunting): my global_id is 4278 2024-03-25T14:21:13.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.360+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c004eb0 con 0x7fd0280a98f0 2024-03-25T14:21:13.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.362+0000 7fd01effd640 10 client.?.objecter ms_handle_connect 0x7fd0281430c0 2024-03-25T14:21:13.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.362+0000 7fd01effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:13.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.384+0000 7fd01effd640 10 client.?.objecter ms_handle_connect 0x7fd02813f710 2024-03-25T14:21:13.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.384+0000 7fd01effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:13.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.397+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391524654 0 0) 0x7fd00c004eb0 con 0x7fd0280a98f0 2024-03-25T14:21:13.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.397+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0281339d0 con 0x7fd0280a98f0 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.397+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd024001710 con 0x7fd0280a98f0 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.397+0000 7fd01effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.397+0000 7fd01effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.397+0000 7fd01effd640 10 monclient(hunting): dump: 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:13.401 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:13.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.397+0000 7fd01effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3935936380 0 0) 0x7fd0281339d0 con 0x7fd0280a98f0 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 >> v1:172.21.15.63:6790/0 conn(0x7fd02813f710 legacy=0x7fd028132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 >> v1:172.21.15.153:6789/0 conn(0x7fd0281430c0 legacy=0x7fd0281332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 10 monclient: _finish_hunting 0 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 1 monclient: found mon.a 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:21:13.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0281370f0 con 0x7fd0280a98f0 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 10 monclient: _finish_auth 0 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 10 monclient: _check_auth_tickets 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.460+0000 7fd01effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 5 monclient: authenticate success, global_id 4278 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 10 monclient: _renew_subs 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd028137340 con 0x7fd0280a98f0 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 10 monclient: _renew_subs 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd028137f00 con 0x7fd0280a98f0 2024-03-25T14:21:13.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.461+0000 7fd0302dd640 1 librados: init done 2024-03-25T14:21:13.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.462+0000 7fd0302dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:13.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.462+0000 7fd0302dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:13.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.462+0000 7fd0302dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:21:13.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.462+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02814f2d0 con 0x7fd0280a98f0 2024-03-25T14:21:13.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.771+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd024003960 con 0x7fd0280a98f0 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.771+0000 7fd01effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.771+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd024003d70 con 0x7fd0280a98f0 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.771+0000 7fd01effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.771+0000 7fd01effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.771+0000 7fd01effd640 10 monclient: dump: 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:13.775 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:13.776 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:13.776 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:13.776 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:13.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.771+0000 7fd02c876640 4 set_mon_vals no callback set 2024-03-25T14:21:13.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.772+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd024003f90 con 0x7fd0280a98f0 2024-03-25T14:21:13.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.773+0000 7fd02d878640 10 monclient: get_auth_request con 0x7fd00c06dd20 auth_method 0 2024-03-25T14:21:13.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.773+0000 7fd01effd640 10 client.4278.objecter ms_handle_connect 0x7fd00c06dd20 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fd024054980 con 0x7fd0280a98f0 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 10 client.4278.objecter ms_dispatch 0x7fd028000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd024050490 con 0x7fd0280a98f0 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:13.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:13.779+0000 7fd01effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:14.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.046+0000 7fd0302dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:14.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.046+0000 7fd0302dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:14.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.046+0000 7fd0302dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:21:14.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.046+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd028137990 con 0x7fd0280a98f0 2024-03-25T14:21:14.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.077+0000 7fd01effd640 1 -- 172.21.15.63:0/475028203 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fd0240523c0 con 0x7fd0280a98f0 2024-03-25T14:21:14.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.077+0000 7fd01effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:14.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.077+0000 7fd01effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:14.080 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:14.080 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:14.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 10 librados: watch_flush enter 2024-03-25T14:21:14.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 10 librados: watch_flush exit 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 10 client.4278.objecter successfully canceled tick 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd00c06dd20 legacy=0x7fd00c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 10 monclient: shutdown 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 >> v1:172.21.15.63:6789/0 conn(0x7fd0280a98f0 legacy=0x7fd028135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 shutdown_connections 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 >> 172.21.15.63:0/475028203 conn(0x7fd0280b7ea0 msgr2=0x7fd028074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 shutdown_connections 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 1 -- 172.21.15.63:0/475028203 wait complete. 2024-03-25T14:21:14.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:14.079+0000 7fd0302dd640 1 librados: shutdown 2024-03-25T14:21:14.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:20.905 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:21.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.017+0000 7fe4a0c14640 10 monclient: get_monmap_and_config 2024-03-25T14:21:21.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.017+0000 7fe4a0c14640 10 monclient: build_initial_monmap 2024-03-25T14:21:21.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.017+0000 7fe4a0c14640 10 monclient: monmap: 2024-03-25T14:21:21.020 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:21.018407+0000 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:21.018407+0000 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.017+0000 7fe4a0c14640 1 Processor -- start 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.017+0000 7fe4a0c14640 1 -- start start 2024-03-25T14:21:21.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.017+0000 7fe4a0c14640 10 monclient: init 2024-03-25T14:21:21.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:21.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 10 monclient(hunting): picked mon.noname-c con 0x7fe49c0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 10 monclient(hunting): picked mon.noname-a con 0x7fe49c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 10 monclient(hunting): picked mon.noname-b con 0x7fe49c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c142520 con 0x7fe49c13f710 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c0b28a0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c0b2a30 con 0x7fe49c1430c0 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:21.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.018+0000 7fe4a0c14640 10 monclient(hunting): authenticate will time out at 1444.163338s 2024-03-25T14:21:21.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.029+0000 7fe499d9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe49c13f710 0x7fe49c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:39672/0 (socket says 172.21.15.63:39672) 2024-03-25T14:21:21.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.029+0000 7fe499d9b640 1 -- 172.21.15.63:0/2340758755 learned_addr learned my addr 172.21.15.63:0/2340758755 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:21.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.042+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252525469 0 0) 0x7fe49c142520 con 0x7fe49c13f710 2024-03-25T14:21:21.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.042+0000 7fe49959a640 10 monclient(hunting): _negotiate 2024-03-25T14:21:21.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.042+0000 7fe49959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:21.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.042+0000 7fe49959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:21.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.042+0000 7fe49959a640 10 monclient(hunting): my global_id is 4281 2024-03-25T14:21:21.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.042+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe484003520 con 0x7fe49c13f710 2024-03-25T14:21:21.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714041632 0 0) 0x7fe484003520 con 0x7fe49c13f710 2024-03-25T14:21:21.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 >> v1:172.21.15.63:6790/0 conn(0x7fe49c1430c0 legacy=0x7fe49c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 >> v1:172.21.15.153:6789/0 conn(0x7fe49c0a98f0 legacy=0x7fe49c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 10 monclient: _finish_hunting 0 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 1 monclient: found mon.noname-a 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c0b44c0 con 0x7fe49c13f710 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 10 monclient: _finish_auth 0 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 10 monclient: _check_auth_tickets 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:21.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe490001690 con 0x7fe49c13f710 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe4a0c14640 5 monclient: authenticate success, global_id 4281 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe49959a640 10 monclient: dump: 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:21.060 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:21.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.054+0000 7fe4a0c14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:21.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4900017f0 con 0x7fe49c13f710 2024-03-25T14:21:21.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe49959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:21.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe49959a640 1 -- 172.21.15.63:0/2340758755 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe490001c00 con 0x7fe49c13f710 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe49959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe49959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe49959a640 10 monclient: dump: 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:21.074 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:21.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe4a0c14640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:21.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.070+0000 7fe4a0c14640 4 set_mon_vals no callback set 2024-03-25T14:21:21.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.071+0000 7fe4a0c14640 10 monclient: shutdown 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.071+0000 7fe4a0c14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.071+0000 7fe4a0c14640 1 -- 172.21.15.63:0/2340758755 >> v1:172.21.15.63:6789/0 conn(0x7fe49c13f710 legacy=0x7fe49c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.071+0000 7fe4a0c14640 1 -- 172.21.15.63:0/2340758755 shutdown_connections 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.071+0000 7fe4a0c14640 1 -- 172.21.15.63:0/2340758755 >> 172.21.15.63:0/2340758755 conn(0x7fe49c0b7ea0 msgr2=0x7fe49c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.071+0000 7fe4a0c14640 1 -- 172.21.15.63:0/2340758755 shutdown_connections 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.071+0000 7fe4a0c14640 1 -- 172.21.15.63:0/2340758755 wait complete. 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.072+0000 7fe4a0c14640 10 monclient: build_initial_monmap 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.072+0000 7fe4a0c14640 10 monclient: monmap: 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:21.077 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 librados: starting msgr at 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 librados: starting objecter 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 Processor -- start 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 -- start start 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 librados: setting wanted keys 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 librados: calling monclient init 2024-03-25T14:21:21.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient: init 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient(hunting): picked mon.noname-b con 0x7fe49c0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient(hunting): picked mon.noname-a con 0x7fe49c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient(hunting): picked mon.noname-c con 0x7fe49c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c0abd70 con 0x7fe49c13f710 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c1d7710 con 0x7fe49c0a98f0 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c1d8340 con 0x7fe49c1430c0 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:21.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.073+0000 7fe4a0c14640 10 monclient(hunting): authenticate will time out at 1444.218424s 2024-03-25T14:21:21.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.078+0000 7fe499d9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe49c13f710 0x7fe49c1d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:39704/0 (socket says 172.21.15.63:39704) 2024-03-25T14:21:21.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.078+0000 7fe499d9b640 1 -- 172.21.15.63:0/1544014784 learned_addr learned my addr 172.21.15.63:0/1544014784 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:21.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.084+0000 7fe48affd640 10 client.?.objecter ms_handle_connect 0x7fe49c13f710 2024-03-25T14:21:21.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.084+0000 7fe48affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:21.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.089+0000 7fe48affd640 10 client.?.objecter ms_handle_connect 0x7fe49c1430c0 2024-03-25T14:21:21.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.089+0000 7fe48affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:21.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.094+0000 7fe48affd640 10 client.?.objecter ms_handle_connect 0x7fe49c0a98f0 2024-03-25T14:21:21.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.094+0000 7fe48affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:21.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.096+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549708713 0 0) 0x7fe49c0abd70 con 0x7fe49c13f710 2024-03-25T14:21:21.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.096+0000 7fe48affd640 10 monclient(hunting): _negotiate 2024-03-25T14:21:21.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.096+0000 7fe48affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:21.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.096+0000 7fe48affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:21.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.096+0000 7fe48affd640 10 monclient(hunting): my global_id is 4284 2024-03-25T14:21:21.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.096+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe47c004eb0 con 0x7fe49c13f710 2024-03-25T14:21:21.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.105+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766436581 0 0) 0x7fe49c1d7710 con 0x7fe49c0a98f0 2024-03-25T14:21:21.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.105+0000 7fe48affd640 10 monclient(hunting): _negotiate 2024-03-25T14:21:21.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.105+0000 7fe48affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:21.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.105+0000 7fe48affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:21.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.105+0000 7fe48affd640 10 monclient(hunting): my global_id is 4297 2024-03-25T14:21:21.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.105+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c0abd70 con 0x7fe49c0a98f0 2024-03-25T14:21:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.107+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314059706 0 0) 0x7fe49c1d8340 con 0x7fe49c1430c0 2024-03-25T14:21:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.107+0000 7fe48affd640 10 monclient(hunting): _negotiate 2024-03-25T14:21:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.107+0000 7fe48affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.107+0000 7fe48affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.107+0000 7fe48affd640 10 monclient(hunting): my global_id is 4283 2024-03-25T14:21:21.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.107+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c1d7710 con 0x7fe49c1430c0 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.117+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 763894073 0 0) 0x7fe49c0abd70 con 0x7fe49c0a98f0 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.117+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe49c1d8340 con 0x7fe49c0a98f0 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.117+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe48c0018a0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.117+0000 7fe48affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.117+0000 7fe48affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.117+0000 7fe48affd640 10 monclient(hunting): dump: 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.120 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.121 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:21.121 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:21.121 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:21.121 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:21.121 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:21.121 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:21.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.117+0000 7fe48affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.128+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258238523 0 0) 0x7fe47c004eb0 con 0x7fe49c13f710 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.128+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe49c0abd70 con 0x7fe49c13f710 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.128+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe490001c00 con 0x7fe49c13f710 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.128+0000 7fe48affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.128+0000 7fe48affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.128+0000 7fe48affd640 10 monclient(hunting): dump: 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:21.131 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.132 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.132 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:21.132 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:21.132 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:21.132 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:21.132 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:21.132 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4161361545 0 0) 0x7fe49c1d8340 con 0x7fe49c0a98f0 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 >> v1:172.21.15.63:6790/0 conn(0x7fe49c1430c0 legacy=0x7fe49c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 >> v1:172.21.15.63:6789/0 conn(0x7fe49c13f710 legacy=0x7fe49c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 10 monclient: _finish_hunting 0 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 1 monclient: found mon.b 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:21.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c1d84d0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 10 monclient: _finish_auth 0 2024-03-25T14:21:21.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 10 monclient: _check_auth_tickets 2024-03-25T14:21:21.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe48affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:21.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe4a0c14640 5 monclient: authenticate success, global_id 4297 2024-03-25T14:21:21.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe4a0c14640 10 monclient: _renew_subs 2024-03-25T14:21:21.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.135+0000 7fe4a0c14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:21.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.136+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c1d41e0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.136+0000 7fe4a0c14640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:21.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.136+0000 7fe4a0c14640 10 monclient: _renew_subs 2024-03-25T14:21:21.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.136+0000 7fe4a0c14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:21.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.136+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c1d5150 con 0x7fe49c0a98f0 2024-03-25T14:21:21.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.136+0000 7fe4a0c14640 1 librados: init done 2024-03-25T14:21:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.137+0000 7fe4a0c14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.137+0000 7fe4a0c14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.137+0000 7fe4a0c14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:21.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.137+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49c0627b0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.411+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe48c001d50 con 0x7fe49c0a98f0 2024-03-25T14:21:21.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.411+0000 7fe48affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.411+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe48c003790 con 0x7fe49c0a98f0 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.411+0000 7fe48affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.411+0000 7fe48affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.411+0000 7fe48affd640 10 monclient: dump: 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:21.415 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:21.416 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:21.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.411+0000 7fe498d99640 4 set_mon_vals no callback set 2024-03-25T14:21:21.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.412+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fe48c0039b0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.413+0000 7fe499d9b640 10 monclient: get_auth_request con 0x7fe47c06df70 auth_method 0 2024-03-25T14:21:21.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.413+0000 7fe48affd640 10 client.4297.objecter ms_handle_connect 0x7fe47c06df70 2024-03-25T14:21:21.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.418+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fe48c053570 con 0x7fe49c0a98f0 2024-03-25T14:21:21.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.418+0000 7fe48affd640 10 client.4297.objecter ms_dispatch 0x7fe49c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:21.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.418+0000 7fe48affd640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:21.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.418+0000 7fe48affd640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:21.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.418+0000 7fe48affd640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-25T14:21:21.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.420+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe48c04eeb0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.420+0000 7fe48affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:21.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.420+0000 7fe48affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:21.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.701+0000 7fe4a0c14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.701+0000 7fe4a0c14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.701+0000 7fe4a0c14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.701+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe49c074cc0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.795+0000 7fe48affd640 1 -- 172.21.15.63:0/1544014784 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fe48c050fb0 con 0x7fe49c0a98f0 2024-03-25T14:21:21.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.796+0000 7fe48affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:21.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.796+0000 7fe48affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:21.799 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:21.799 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 10 librados: watch_flush enter 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 10 librados: watch_flush exit 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 10 client.4297.objecter successfully canceled tick 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 >> v1:172.21.15.153:6800/1688255552 conn(0x7fe47c06df70 legacy=0x7fe47c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 10 monclient: shutdown 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 >> v1:172.21.15.153:6789/0 conn(0x7fe49c0a98f0 legacy=0x7fe49c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 shutdown_connections 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 >> 172.21.15.63:0/1544014784 conn(0x7fe49c0b7ea0 msgr2=0x7fe49c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.798+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 shutdown_connections 2024-03-25T14:21:21.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.799+0000 7fe4a0c14640 1 -- 172.21.15.63:0/1544014784 wait complete. 2024-03-25T14:21:21.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:21.799+0000 7fe4a0c14640 1 librados: shutdown 2024-03-25T14:21:21.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:26.054 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:04.568 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.059 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:04.572 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.091 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:04.605 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.111 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:04.571 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.117 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:04.578 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.120 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:04.634 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.127 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:04.641 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.144 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:04.605 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:26.152 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:04.666 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:28.621 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:28.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.739+0000 7fbc190e2640 10 monclient: get_monmap_and_config 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.739+0000 7fbc190e2640 10 monclient: build_initial_monmap 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.739+0000 7fbc190e2640 10 monclient: monmap: 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:28.741059+0000 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:28.741059+0000 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:28.743 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.740+0000 7fbc190e2640 1 Processor -- start 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.740+0000 7fbc190e2640 1 -- start start 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.740+0000 7fbc190e2640 10 monclient: init 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 10 monclient(hunting): picked mon.noname-c con 0x7fbc140b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 10 monclient(hunting): picked mon.noname-b con 0x7fbc140b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1414be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc140af940 con 0x7fbc1414be90 2024-03-25T14:21:28.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc140b0a70 con 0x7fbc140b4d60 2024-03-25T14:21:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc140b0d50 con 0x7fbc140b1730 2024-03-25T14:21:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:28.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.741+0000 7fbc190e2640 10 monclient(hunting): authenticate will time out at 1451.886286s 2024-03-25T14:21:28.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.745+0000 7fbc1359e640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fbc1414be90 0x7fbc1414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44190/0 (socket says 172.21.15.63:44190) 2024-03-25T14:21:28.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.745+0000 7fbc1359e640 1 -- 172.21.15.63:0/2139309764 learned_addr learned my addr 172.21.15.63:0/2139309764 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:28.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.753+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176483164 0 0) 0x7fbc140af940 con 0x7fbc1414be90 2024-03-25T14:21:28.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.753+0000 7fbc1259c640 10 monclient(hunting): _negotiate 2024-03-25T14:21:28.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.753+0000 7fbc1259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:28.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.753+0000 7fbc1259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:28.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.753+0000 7fbc1259c640 10 monclient(hunting): my global_id is 4287 2024-03-25T14:21:28.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.753+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc00003520 con 0x7fbc1414be90 2024-03-25T14:21:28.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589863260 0 0) 0x7fbc140b0a70 con 0x7fbc140b4d60 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): _negotiate 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): my global_id is 4300 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc140af940 con 0x7fbc140b4d60 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562954632 0 0) 0x7fbc140b0d50 con 0x7fbc140b1730 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): _negotiate 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 10 monclient(hunting): my global_id is 4286 2024-03-25T14:21:28.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.771+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc140b0a70 con 0x7fbc140b1730 2024-03-25T14:21:28.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1092165014 0 0) 0x7fbc00003520 con 0x7fbc1414be90 2024-03-25T14:21:28.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 >> v1:172.21.15.63:6790/0 conn(0x7fbc140b1730 legacy=0x7fbc140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 >> v1:172.21.15.153:6789/0 conn(0x7fbc140b4d60 legacy=0x7fbc1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 10 monclient: _finish_hunting 0 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 1 monclient: found mon.noname-a 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc140b1030 con 0x7fbc1414be90 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 10 monclient: _finish_auth 0 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 10 monclient: _check_auth_tickets 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbc0c001a10 con 0x7fbc1414be90 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc190e2640 5 monclient: authenticate success, global_id 4287 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc190e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:28.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc1259c640 10 monclient: dump: 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:28.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.776+0000 7fbc190e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc0c001b70 con 0x7fbc1414be90 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc1259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc1259c640 1 -- 172.21.15.63:0/2139309764 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbc0c001f80 con 0x7fbc1414be90 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc1259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc1259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc1259c640 10 monclient: dump: 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.804 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc190e2640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:28.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.801+0000 7fbc190e2640 4 set_mon_vals no callback set 2024-03-25T14:21:28.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.802+0000 7fbc190e2640 10 monclient: shutdown 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.802+0000 7fbc190e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.802+0000 7fbc190e2640 1 -- 172.21.15.63:0/2139309764 >> v1:172.21.15.63:6789/0 conn(0x7fbc1414be90 legacy=0x7fbc1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.802+0000 7fbc190e2640 1 -- 172.21.15.63:0/2139309764 shutdown_connections 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.802+0000 7fbc190e2640 1 -- 172.21.15.63:0/2139309764 >> 172.21.15.63:0/2139309764 conn(0x7fbc140617a0 msgr2=0x7fbc14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.802+0000 7fbc190e2640 1 -- 172.21.15.63:0/2139309764 shutdown_connections 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.802+0000 7fbc190e2640 1 -- 172.21.15.63:0/2139309764 wait complete. 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 10 monclient: build_initial_monmap 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 10 monclient: monmap: 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:28.808 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 1 librados: starting msgr at 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 1 librados: starting objecter 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 1 Processor -- start 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 1 -- start start 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 1 librados: setting wanted keys 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 1 librados: calling monclient init 2024-03-25T14:21:28.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.803+0000 7fbc190e2640 10 monclient: init 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 10 monclient(hunting): picked mon.noname-b con 0x7fbc140b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 10 monclient(hunting): picked mon.noname-c con 0x7fbc140b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1414be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc14141950 con 0x7fbc1414be90 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc141d7940 con 0x7fbc140b1730 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc141d8570 con 0x7fbc140b4d60 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:28.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.804+0000 7fbc190e2640 10 monclient(hunting): authenticate will time out at 1451.949305s 2024-03-25T14:21:28.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.819+0000 7fbc12d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbc140b1730 0x7fbc141d7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:42282/0 (socket says 172.21.15.63:42282) 2024-03-25T14:21:28.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.819+0000 7fbc12d9d640 1 -- 172.21.15.63:0/160460183 learned_addr learned my addr 172.21.15.63:0/160460183 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:28.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.819+0000 7fbc0b7fe640 10 client.?.objecter ms_handle_connect 0x7fbc140b1730 2024-03-25T14:21:28.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.819+0000 7fbc0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.822+0000 7fbc0b7fe640 10 client.?.objecter ms_handle_connect 0x7fbc1414be90 2024-03-25T14:21:28.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.822+0000 7fbc0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:28.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.824+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251579378 0 0) 0x7fbc141d7940 con 0x7fbc140b1730 2024-03-25T14:21:28.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.824+0000 7fbc0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:28.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.824+0000 7fbc0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:28.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.824+0000 7fbc0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:28.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.824+0000 7fbc0b7fe640 10 monclient(hunting): my global_id is 4303 2024-03-25T14:21:28.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.824+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf4004eb0 con 0x7fbc140b1730 2024-03-25T14:21:28.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.832+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901954040 0 0) 0x7fbc14141950 con 0x7fbc1414be90 2024-03-25T14:21:28.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.832+0000 7fbc0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:28.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.832+0000 7fbc0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:28.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.832+0000 7fbc0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:28.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.832+0000 7fbc0b7fe640 10 monclient(hunting): my global_id is 4290 2024-03-25T14:21:28.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.832+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc141d7940 con 0x7fbc1414be90 2024-03-25T14:21:28.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.833+0000 7fbc0b7fe640 10 client.?.objecter ms_handle_connect 0x7fbc140b4d60 2024-03-25T14:21:28.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.833+0000 7fbc0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:28.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.837+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181689260 0 0) 0x7fbc141d8570 con 0x7fbc140b4d60 2024-03-25T14:21:28.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.837+0000 7fbc0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:28.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.837+0000 7fbc0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:28.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.837+0000 7fbc0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:28.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.837+0000 7fbc0b7fe640 10 monclient(hunting): my global_id is 4289 2024-03-25T14:21:28.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.837+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc14141950 con 0x7fbc140b4d60 2024-03-25T14:21:28.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.841+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077651670 0 0) 0x7fbbf4004eb0 con 0x7fbc140b1730 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.841+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc141d8570 con 0x7fbc140b1730 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.841+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbc04001a30 con 0x7fbc140b1730 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.841+0000 7fbc0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.841+0000 7fbc0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.841+0000 7fbc0b7fe640 10 monclient(hunting): dump: 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:28.844 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:28.845 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:28.845 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:28.845 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:28.845 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.841+0000 7fbc0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.845+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637727689 0 0) 0x7fbc141d7940 con 0x7fbc1414be90 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.845+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf4004eb0 con 0x7fbc1414be90 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.845+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbc0c001e80 con 0x7fbc1414be90 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.845+0000 7fbc0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.845+0000 7fbc0b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.845+0000 7fbc0b7fe640 10 monclient(hunting): dump: 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:28.848 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:28.849 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:28.849 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:28.849 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:28.849 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3664943194 0 0) 0x7fbc141d8570 con 0x7fbc140b1730 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 >> v1:172.21.15.63:6790/0 conn(0x7fbc140b4d60 legacy=0x7fbc14142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 >> v1:172.21.15.63:6789/0 conn(0x7fbc1414be90 legacy=0x7fbc141e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 1 monclient: found mon.b 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:28.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc14138840 con 0x7fbc140b1730 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 5 monclient: authenticate success, global_id 4303 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 10 monclient: _renew_subs 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc14142f70 con 0x7fbc140b1730 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 10 monclient: _renew_subs 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc14137da0 con 0x7fbc140b1730 2024-03-25T14:21:28.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.847+0000 7fbc190e2640 1 librados: init done 2024-03-25T14:21:28.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.848+0000 7fbc190e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:28.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.848+0000 7fbc190e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:28.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.848+0000 7fbc190e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:28.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:28.848+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc141d7ad0 con 0x7fbc140b1730 2024-03-25T14:21:29.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.136+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc04001c70 con 0x7fbc140b1730 2024-03-25T14:21:29.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.136+0000 7fbc0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:29.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.136+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbc04003a60 con 0x7fbc140b1730 2024-03-25T14:21:29.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.136+0000 7fbc0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:29.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.136+0000 7fbc0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.136+0000 7fbc0b7fe640 10 monclient: dump: 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.136+0000 7fbc11d9b640 4 set_mon_vals no callback set 2024-03-25T14:21:29.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.137+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fbc04003c80 con 0x7fbc140b1730 2024-03-25T14:21:29.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.138+0000 7fbc0bfff640 10 monclient: get_auth_request con 0x7fbbf406dee0 auth_method 0 2024-03-25T14:21:29.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.138+0000 7fbc0b7fe640 10 client.4303.objecter ms_handle_connect 0x7fbbf406dee0 2024-03-25T14:21:29.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.142+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fbc04054890 con 0x7fbc140b1730 2024-03-25T14:21:29.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.142+0000 7fbc0b7fe640 10 client.4303.objecter ms_dispatch 0x7fbc14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:29.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.142+0000 7fbc0b7fe640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:29.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.142+0000 7fbc0b7fe640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:29.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.142+0000 7fbc0b7fe640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-25T14:21:29.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.143+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc040503a0 con 0x7fbc140b1730 2024-03-25T14:21:29.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.143+0000 7fbc0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:29.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.143+0000 7fbc0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:29.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.371+0000 7fbc190e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:29.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.371+0000 7fbc190e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:29.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.371+0000 7fbc190e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:29.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.371+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc141389d0 con 0x7fbc140b1730 2024-03-25T14:21:29.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.417+0000 7fbc0b7fe640 1 -- 172.21.15.63:0/160460183 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fbc040522d0 con 0x7fbc140b1730 2024-03-25T14:21:29.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.417+0000 7fbc0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:29.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.417+0000 7fbc0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:29.421 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:29.421 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:29.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.419+0000 7fbc190e2640 10 librados: watch_flush enter 2024-03-25T14:21:29.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.419+0000 7fbc190e2640 10 librados: watch_flush exit 2024-03-25T14:21:29.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.419+0000 7fbc190e2640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-25T14:21:29.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 10 client.4303.objecter successfully canceled tick 2024-03-25T14:21:29.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 >> v1:172.21.15.153:6800/1688255552 conn(0x7fbbf406dee0 legacy=0x7fbbf40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:29.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 10 monclient: shutdown 2024-03-25T14:21:29.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:29.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 >> v1:172.21.15.153:6789/0 conn(0x7fbc140b1730 legacy=0x7fbc141d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:29.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 shutdown_connections 2024-03-25T14:21:29.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 >> 172.21.15.63:0/160460183 conn(0x7fbc140617a0 msgr2=0x7fbc140b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:29.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 shutdown_connections 2024-03-25T14:21:29.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 1 -- 172.21.15.63:0/160460183 wait complete. 2024-03-25T14:21:29.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:29.420+0000 7fbc190e2640 1 librados: shutdown 2024-03-25T14:21:29.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:36.241 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:36.352 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.350+0000 7f3ab4300640 10 monclient: get_monmap_and_config 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.350+0000 7f3ab4300640 10 monclient: build_initial_monmap 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.350+0000 7f3ab4300640 10 monclient: monmap: 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:36.350810+0000 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:36.350810+0000 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:36.353 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.351+0000 7f3ab4300640 1 Processor -- start 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.351+0000 7f3ab4300640 1 -- start start 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.351+0000 7f3ab4300640 10 monclient: init 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 10 monclient(hunting): picked mon.noname-c con 0x7f3aac0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 10 monclient(hunting): picked mon.noname-b con 0x7f3aac13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:36.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 10 monclient(hunting): picked mon.noname-a con 0x7f3aac1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:36.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac142520 con 0x7f3aac1430c0 2024-03-25T14:21:36.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac0b28a0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac0b2a30 con 0x7f3aac13f710 2024-03-25T14:21:36.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:36.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.352+0000 7f3ab4300640 10 monclient(hunting): authenticate will time out at 1459.496075s 2024-03-25T14:21:36.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.359+0000 7f3ab289d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f3aac1430c0 0x7f3aac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:53154/0 (socket says 172.21.15.63:53154) 2024-03-25T14:21:36.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.359+0000 7f3ab289d640 1 -- 172.21.15.63:0/3284537983 learned_addr learned my addr 172.21.15.63:0/3284537983 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:36.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.373+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630373226 0 0) 0x7f3aac142520 con 0x7f3aac1430c0 2024-03-25T14:21:36.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.373+0000 7f3ab109a640 10 monclient(hunting): _negotiate 2024-03-25T14:21:36.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.373+0000 7f3ab109a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:36.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.373+0000 7f3ab109a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:36.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.373+0000 7f3ab109a640 10 monclient(hunting): my global_id is 4293 2024-03-25T14:21:36.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.373+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a98003520 con 0x7f3aac1430c0 2024-03-25T14:21:36.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.376+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448978200 0 0) 0x7f3aac0b28a0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.376+0000 7f3ab109a640 10 monclient(hunting): _negotiate 2024-03-25T14:21:36.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.376+0000 7f3ab109a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:36.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.376+0000 7f3ab109a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:36.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.376+0000 7f3ab109a640 10 monclient(hunting): my global_id is 4306 2024-03-25T14:21:36.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.376+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac142520 con 0x7f3aac0a98f0 2024-03-25T14:21:36.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.385+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137282792 0 0) 0x7f3aac0b2a30 con 0x7f3aac13f710 2024-03-25T14:21:36.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.385+0000 7f3ab109a640 10 monclient(hunting): _negotiate 2024-03-25T14:21:36.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.385+0000 7f3ab109a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:36.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.385+0000 7f3ab109a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:36.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.385+0000 7f3ab109a640 10 monclient(hunting): my global_id is 4292 2024-03-25T14:21:36.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.385+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac0b28a0 con 0x7f3aac13f710 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.387+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3647989934 0 0) 0x7f3a98003520 con 0x7f3aac1430c0 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 >> v1:172.21.15.63:6790/0 conn(0x7f3aac13f710 legacy=0x7f3aac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 >> v1:172.21.15.153:6789/0 conn(0x7f3aac0a98f0 legacy=0x7f3aac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 10 monclient: _finish_hunting 0 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 1 monclient: found mon.noname-a 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0b44c0 con 0x7f3aac1430c0 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 10 monclient: _finish_auth 0 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 10 monclient: _check_auth_tickets 2024-03-25T14:21:36.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3aa4001690 con 0x7f3aac1430c0 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab109a640 10 monclient: dump: 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:36.391 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:36.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab4300640 5 monclient: authenticate success, global_id 4293 2024-03-25T14:21:36.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.388+0000 7f3ab4300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:36.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3aa40017f0 con 0x7f3aac1430c0 2024-03-25T14:21:36.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab109a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:36.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab109a640 1 -- 172.21.15.63:0/3284537983 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3aa4001c00 con 0x7f3aac1430c0 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab4300640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab4300640 4 set_mon_vals no callback set 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab109a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab109a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.402+0000 7f3ab109a640 10 monclient: dump: 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:36.405 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:36.406 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:36.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.403+0000 7f3ab4300640 10 monclient: shutdown 2024-03-25T14:21:36.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.403+0000 7f3ab4300640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:36.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.403+0000 7f3ab4300640 1 -- 172.21.15.63:0/3284537983 >> v1:172.21.15.63:6789/0 conn(0x7f3aac1430c0 legacy=0x7f3aac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:36.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.403+0000 7f3ab4300640 1 -- 172.21.15.63:0/3284537983 shutdown_connections 2024-03-25T14:21:36.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.403+0000 7f3ab4300640 1 -- 172.21.15.63:0/3284537983 >> 172.21.15.63:0/3284537983 conn(0x7f3aac0b7ea0 msgr2=0x7f3aac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.403+0000 7f3ab4300640 1 -- 172.21.15.63:0/3284537983 shutdown_connections 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.403+0000 7f3ab4300640 1 -- 172.21.15.63:0/3284537983 wait complete. 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.404+0000 7f3ab4300640 10 monclient: build_initial_monmap 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.404+0000 7f3ab4300640 10 monclient: monmap: 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:36.408 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:36.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.405+0000 7f3ab4300640 1 librados: starting msgr at 2024-03-25T14:21:36.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.405+0000 7f3ab4300640 1 librados: starting objecter 2024-03-25T14:21:36.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.405+0000 7f3ab4300640 1 Processor -- start 2024-03-25T14:21:36.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.405+0000 7f3ab4300640 1 -- start start 2024-03-25T14:21:36.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.405+0000 7f3ab4300640 1 librados: setting wanted keys 2024-03-25T14:21:36.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.405+0000 7f3ab4300640 1 librados: calling monclient init 2024-03-25T14:21:36.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.405+0000 7f3ab4300640 10 monclient: init 2024-03-25T14:21:36.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:36.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:21:36.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 10 monclient(hunting): picked mon.noname-c con 0x7f3aac0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:36.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 10 monclient(hunting): picked mon.noname-b con 0x7f3aac13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:36.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 10 monclient(hunting): picked mon.noname-a con 0x7f3aac1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:36.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac136a50 con 0x7f3aac1430c0 2024-03-25T14:21:36.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac0b79c0 con 0x7f3aac13f710 2024-03-25T14:21:36.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac0b7ca0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:36.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.406+0000 7f3ab4300640 10 monclient(hunting): authenticate will time out at 1459.549742s 2024-03-25T14:21:36.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.412+0000 7f3ab209c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f3aac0a98f0 0x7f3aac137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:56274/0 (socket says 172.21.15.63:56274) 2024-03-25T14:21:36.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.412+0000 7f3ab209c640 1 -- 172.21.15.63:0/848962406 learned_addr learned my addr 172.21.15.63:0/848962406 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:36.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.413+0000 7f3aa2ffd640 10 client.?.objecter ms_handle_connect 0x7f3aac0a98f0 2024-03-25T14:21:36.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.413+0000 7f3aa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:36.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.418+0000 7f3aa2ffd640 10 client.?.objecter ms_handle_connect 0x7f3aac13f710 2024-03-25T14:21:36.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.418+0000 7f3aa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:36.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.419+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079382297 0 0) 0x7f3aac0b7ca0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.419+0000 7f3aa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:21:36.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.419+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:36.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.419+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:36.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.419+0000 7f3aa2ffd640 10 monclient(hunting): my global_id is 4295 2024-03-25T14:21:36.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.419+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a90004eb0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.429+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625635642 0 0) 0x7f3aac0b79c0 con 0x7f3aac13f710 2024-03-25T14:21:36.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.429+0000 7f3aa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:21:36.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.429+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:36.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.429+0000 7f3aa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:36.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.429+0000 7f3aa2ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T14:21:36.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.429+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac0b7ca0 con 0x7f3aac13f710 2024-03-25T14:21:36.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.430+0000 7f3aa2ffd640 10 client.?.objecter ms_handle_connect 0x7f3aac1430c0 2024-03-25T14:21:36.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.430+0000 7f3aa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.437+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456936171 0 0) 0x7f3a90004eb0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.437+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aac0b79c0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.437+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3aa8001db0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.437+0000 7f3aa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.437+0000 7f3aa2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.437+0000 7f3aa2ffd640 10 monclient(hunting): dump: 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:36.439 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:36.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.437+0000 7f3aa2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:36.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.442+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2641314327 0 0) 0x7f3aac0b7ca0 con 0x7f3aac13f710 2024-03-25T14:21:36.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.442+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a90004eb0 con 0x7f3aac13f710 2024-03-25T14:21:36.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.442+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3a9c0018b0 con 0x7f3aac13f710 2024-03-25T14:21:36.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.442+0000 7f3aa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:36.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.442+0000 7f3aa2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.442+0000 7f3aa2ffd640 10 monclient(hunting): dump: 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:36.445 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:36.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2314420415 0 0) 0x7f3aac0b79c0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 >> v1:172.21.15.153:6789/0 conn(0x7f3aac13f710 legacy=0x7f3aac135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:36.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 >> v1:172.21.15.63:6789/0 conn(0x7f3aac1430c0 legacy=0x7f3aac1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:36.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:21:36.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 1 monclient: found mon.c 2024-03-25T14:21:36.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:36.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac136ca0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 10 monclient: _finish_auth 0 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3aa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 5 monclient: authenticate success, global_id 4295 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 10 monclient: _renew_subs 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aac136f80 con 0x7f3aac0a98f0 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 10 monclient: _renew_subs 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac1360e0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.445+0000 7f3ab4300640 1 librados: init done 2024-03-25T14:21:36.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.446+0000 7f3ab4300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:36.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.446+0000 7f3ab4300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:36.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.446+0000 7f3ab4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:36.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.446+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac0627b0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.478+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3aa8001ff0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.478+0000 7f3aa2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.478+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3aa8003de0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.478+0000 7f3aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.478+0000 7f3aa2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.478+0000 7f3aa2ffd640 10 monclient: dump: 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:36.481 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:36.482 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:36.482 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:36.482 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:36.482 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:36.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.478+0000 7f3ab0899640 4 set_mon_vals no callback set 2024-03-25T14:21:36.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.479+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f3aa8003fc0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.480+0000 7f3ab189b640 10 monclient: get_auth_request con 0x7f3a9006de90 auth_method 0 2024-03-25T14:21:36.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.480+0000 7f3aa2ffd640 10 client.4295.objecter ms_handle_connect 0x7f3a9006de90 2024-03-25T14:21:36.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.483+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f3aa8054bb0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.483+0000 7f3aa2ffd640 10 client.4295.objecter ms_dispatch 0x7f3aac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:36.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.483+0000 7f3aa2ffd640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:36.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.483+0000 7f3aa2ffd640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:36.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.483+0000 7f3aa2ffd640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-25T14:21:36.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.792+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3aa80506c0 con 0x7f3aac0a98f0 2024-03-25T14:21:36.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.792+0000 7f3aa2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:36.794 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:36.792+0000 7f3aa2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.043+0000 7f3ab4300640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.044+0000 7f3ab4300640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.044+0000 7f3ab4300640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.044+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3aac0008d0 con 0x7f3aac0a98f0 2024-03-25T14:21:37.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.089+0000 7f3aa2ffd640 1 -- 172.21.15.63:0/848962406 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f3aa80525f0 con 0x7f3aac0a98f0 2024-03-25T14:21:37.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.089+0000 7f3aa2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:37.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.089+0000 7f3aa2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:37.091 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:37.092 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:37.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.091+0000 7f3ab4300640 10 librados: watch_flush enter 2024-03-25T14:21:37.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.091+0000 7f3ab4300640 10 librados: watch_flush exit 2024-03-25T14:21:37.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-25T14:21:37.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 10 client.4295.objecter successfully canceled tick 2024-03-25T14:21:37.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 >> v1:172.21.15.153:6800/1688255552 conn(0x7f3a9006de90 legacy=0x7f3a90070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:37.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 10 monclient: shutdown 2024-03-25T14:21:37.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:37.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 >> v1:172.21.15.63:6790/0 conn(0x7f3aac0a98f0 legacy=0x7f3aac137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:37.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 shutdown_connections 2024-03-25T14:21:37.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 >> 172.21.15.63:0/848962406 conn(0x7f3aac0b7ea0 msgr2=0x7f3aac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:37.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 shutdown_connections 2024-03-25T14:21:37.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 1 -- 172.21.15.63:0/848962406 wait complete. 2024-03-25T14:21:37.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:37.092+0000 7f3ab4300640 1 librados: shutdown 2024-03-25T14:21:37.106 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:43.913 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:44.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.027+0000 7f31039c9640 10 monclient: get_monmap_and_config 2024-03-25T14:21:44.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.027+0000 7f31039c9640 10 monclient: build_initial_monmap 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.027+0000 7f31039c9640 10 monclient: monmap: 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:44.028767+0000 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:44.028767+0000 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:44.031 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.028+0000 7f31039c9640 1 Processor -- start 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.028+0000 7f31039c9640 1 -- start start 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.028+0000 7f31039c9640 10 monclient: init 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 10 monclient(hunting): picked mon.noname-c con 0x7f30fc0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 10 monclient(hunting): picked mon.noname-a con 0x7f30fc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 10 monclient(hunting): picked mon.noname-b con 0x7f30fc1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc142520 con 0x7f30fc13f710 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc0b28a0 con 0x7f30fc0a98f0 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc0b2a30 con 0x7f30fc1430c0 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:44.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.029+0000 7f31039c9640 10 monclient(hunting): authenticate will time out at 1467.174196s 2024-03-25T14:21:44.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.033+0000 7f3101f66640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f30fc1430c0 0x7f30fc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:51164/0 (socket says 172.21.15.63:51164) 2024-03-25T14:21:44.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.033+0000 7f3101f66640 1 -- 172.21.15.63:0/4217119991 learned_addr learned my addr 172.21.15.63:0/4217119991 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:44.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.048+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706282633 0 0) 0x7f30fc0b28a0 con 0x7f30fc0a98f0 2024-03-25T14:21:44.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.048+0000 7f30f3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:21:44.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.048+0000 7f30f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:44.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.048+0000 7f30f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:44.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.048+0000 7f30f3fff640 10 monclient(hunting): my global_id is 4312 2024-03-25T14:21:44.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.048+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e8003520 con 0x7f30fc0a98f0 2024-03-25T14:21:44.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.050+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008802856 0 0) 0x7f30fc142520 con 0x7f30fc13f710 2024-03-25T14:21:44.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.050+0000 7f30f3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:21:44.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.050+0000 7f30f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:44.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.050+0000 7f30f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:44.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.050+0000 7f30f3fff640 10 monclient(hunting): my global_id is 4299 2024-03-25T14:21:44.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.050+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30fc0b28a0 con 0x7f30fc13f710 2024-03-25T14:21:44.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.055+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804427868 0 0) 0x7f30fc0b2a30 con 0x7f30fc1430c0 2024-03-25T14:21:44.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.055+0000 7f30f3fff640 10 monclient(hunting): _negotiate 2024-03-25T14:21:44.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.055+0000 7f30f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:44.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.055+0000 7f30f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:44.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.055+0000 7f30f3fff640 10 monclient(hunting): my global_id is 4298 2024-03-25T14:21:44.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.055+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30fc142520 con 0x7f30fc1430c0 2024-03-25T14:21:44.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903946461 0 0) 0x7f30e8003520 con 0x7f30fc0a98f0 2024-03-25T14:21:44.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 >> v1:172.21.15.63:6790/0 conn(0x7f30fc1430c0 legacy=0x7f30fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:44.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 >> v1:172.21.15.63:6789/0 conn(0x7f30fc13f710 legacy=0x7f30fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:44.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 10 monclient: _finish_hunting 0 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 1 monclient: found mon.noname-c 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0b44c0 con 0x7f30fc0a98f0 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 10 monclient: _finish_auth 0 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 10 monclient: _check_auth_tickets 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f30f8001690 con 0x7f30fc0a98f0 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f30f3fff640 10 monclient: dump: 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:44.067 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f31039c9640 5 monclient: authenticate success, global_id 4312 2024-03-25T14:21:44.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.063+0000 7f31039c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30f80017f0 con 0x7f30fc0a98f0 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f30f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f30f3fff640 1 -- 172.21.15.63:0/4217119991 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f30f8001c00 con 0x7f30fc0a98f0 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f30f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f30f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f30f3fff640 10 monclient: dump: 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:44.075 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:44.076 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:44.076 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:44.076 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f31039c9640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:44.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.072+0000 7f31039c9640 4 set_mon_vals no callback set 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.073+0000 7f31039c9640 10 monclient: shutdown 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.073+0000 7f31039c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.073+0000 7f31039c9640 1 -- 172.21.15.63:0/4217119991 >> v1:172.21.15.153:6789/0 conn(0x7f30fc0a98f0 legacy=0x7f30fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.073+0000 7f31039c9640 1 -- 172.21.15.63:0/4217119991 shutdown_connections 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.073+0000 7f31039c9640 1 -- 172.21.15.63:0/4217119991 >> 172.21.15.63:0/4217119991 conn(0x7f30fc0b7ea0 msgr2=0x7f30fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.073+0000 7f31039c9640 1 -- 172.21.15.63:0/4217119991 shutdown_connections 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.073+0000 7f31039c9640 1 -- 172.21.15.63:0/4217119991 wait complete. 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 10 monclient: build_initial_monmap 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 10 monclient: monmap: 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:44.078 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 1 librados: starting msgr at 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 1 librados: starting objecter 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 1 Processor -- start 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 1 -- start start 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 1 librados: setting wanted keys 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 1 librados: calling monclient init 2024-03-25T14:21:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.074+0000 7f31039c9640 10 monclient: init 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 10 monclient(hunting): picked mon.noname-b con 0x7f30fc0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 10 monclient(hunting): picked mon.noname-c con 0x7f30fc13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 10 monclient(hunting): picked mon.noname-a con 0x7f30fc1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc139a70 con 0x7f30fc1430c0 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc136b50 con 0x7f30fc0a98f0 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc136e30 con 0x7f30fc13f710 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.075+0000 7f31039c9640 10 monclient(hunting): authenticate will time out at 1467.219618s 2024-03-25T14:21:44.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.084+0000 7f3101f66640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f30fc1430c0 0x7f30fc1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52500/0 (socket says 172.21.15.63:52500) 2024-03-25T14:21:44.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.084+0000 7f3101f66640 1 -- 172.21.15.63:0/430765452 learned_addr learned my addr 172.21.15.63:0/430765452 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:44.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.085+0000 7f30f1ffb640 10 client.?.objecter ms_handle_connect 0x7f30fc1430c0 2024-03-25T14:21:44.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.085+0000 7f30f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:44.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.088+0000 7f30f1ffb640 10 client.?.objecter ms_handle_connect 0x7f30fc13f710 2024-03-25T14:21:44.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.088+0000 7f30f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:44.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.094+0000 7f30f1ffb640 10 client.?.objecter ms_handle_connect 0x7f30fc0a98f0 2024-03-25T14:21:44.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.094+0000 7f30f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:44.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.097+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164368980 0 0) 0x7f30fc139a70 con 0x7f30fc1430c0 2024-03-25T14:21:44.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.097+0000 7f30f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:21:44.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.097+0000 7f30f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:44.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.097+0000 7f30f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:44.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.097+0000 7f30f1ffb640 10 monclient(hunting): my global_id is 4302 2024-03-25T14:21:44.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.097+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e0004eb0 con 0x7f30fc1430c0 2024-03-25T14:21:44.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.099+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891889337 0 0) 0x7f30fc136e30 con 0x7f30fc13f710 2024-03-25T14:21:44.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.099+0000 7f30f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:21:44.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.099+0000 7f30f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:44.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.099+0000 7f30f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:44.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.099+0000 7f30f1ffb640 10 monclient(hunting): my global_id is 4301 2024-03-25T14:21:44.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.099+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30fc139a70 con 0x7f30fc13f710 2024-03-25T14:21:44.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.112+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635062279 0 0) 0x7f30fc136b50 con 0x7f30fc0a98f0 2024-03-25T14:21:44.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.112+0000 7f30f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:21:44.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.112+0000 7f30f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:44.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.112+0000 7f30f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:44.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.112+0000 7f30f1ffb640 10 monclient(hunting): my global_id is 4315 2024-03-25T14:21:44.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.112+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30fc136e30 con 0x7f30fc0a98f0 2024-03-25T14:21:44.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.113+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336667117 0 0) 0x7f30e0004eb0 con 0x7f30fc1430c0 2024-03-25T14:21:44.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.113+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30fc136b50 con 0x7f30fc1430c0 2024-03-25T14:21:44.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f30f4001bd0 con 0x7f30fc1430c0 2024-03-25T14:21:44.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:44.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 10 monclient(hunting): dump: 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040295127 0 0) 0x7f30fc139a70 con 0x7f30fc13f710 2024-03-25T14:21:44.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e0004eb0 con 0x7f30fc13f710 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f30ec0018f0 con 0x7f30fc13f710 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.114+0000 7f30f1ffb640 10 monclient(hunting): dump: 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:44.119 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.127+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1020041580 0 0) 0x7f30fc136e30 con 0x7f30fc0a98f0 2024-03-25T14:21:44.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.127+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30fc139a70 con 0x7f30fc0a98f0 2024-03-25T14:21:44.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.127+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f30f8001b00 con 0x7f30fc0a98f0 2024-03-25T14:21:44.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.127+0000 7f30f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:44.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.127+0000 7f30f1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:44.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.127+0000 7f30f1ffb640 10 monclient(hunting): dump: 2024-03-25T14:21:44.130 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:44.131 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1984251718 0 0) 0x7f30e0004eb0 con 0x7f30fc13f710 2024-03-25T14:21:44.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 >> v1:172.21.15.153:6789/0 conn(0x7f30fc0a98f0 legacy=0x7f30fc137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:44.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 >> v1:172.21.15.63:6789/0 conn(0x7f30fc1430c0 legacy=0x7f30fc1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 1 monclient: found mon.c 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc137110 con 0x7f30fc13f710 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.135+0000 7f30f1ffb640 10 monclient: _finish_auth 0 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f30f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f30f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 5 monclient: authenticate success, global_id 4301 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 10 monclient: _renew_subs 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30fc0b7890 con 0x7f30fc13f710 2024-03-25T14:21:44.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:44.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 10 monclient: _renew_subs 2024-03-25T14:21:44.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:44.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30fc135f60 con 0x7f30fc13f710 2024-03-25T14:21:44.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.136+0000 7f31039c9640 1 librados: init done 2024-03-25T14:21:44.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.137+0000 7f31039c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:44.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.137+0000 7f31039c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:44.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.137+0000 7f31039c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:44.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.137+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30fc13d010 con 0x7f30fc13f710 2024-03-25T14:21:44.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.456+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30ec001d60 con 0x7f30fc13f710 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.456+0000 7f30f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.456+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f30ec003790 con 0x7f30fc13f710 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.456+0000 7f30f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.456+0000 7f30f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.456+0000 7f30f1ffb640 10 monclient: dump: 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:44.460 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:44.461 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:44.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.456+0000 7f30f37fe640 4 set_mon_vals no callback set 2024-03-25T14:21:44.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.457+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f30ec0197a0 con 0x7f30fc13f710 2024-03-25T14:21:44.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.458+0000 7f3101765640 10 monclient: get_auth_request con 0x7f30e006e0a0 auth_method 0 2024-03-25T14:21:44.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.458+0000 7f30f1ffb640 10 client.4301.objecter ms_handle_connect 0x7f30e006e0a0 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.463+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f30ec053510 con 0x7f30fc13f710 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.464+0000 7f30f1ffb640 10 client.4301.objecter ms_dispatch 0x7f30fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.464+0000 7f30f1ffb640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.464+0000 7f30f1ffb640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.464+0000 7f30f1ffb640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.464+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30ec04ee50 con 0x7f30fc13f710 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.464+0000 7f30f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:44.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.464+0000 7f30f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:44.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.713+0000 7f31039c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:44.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.713+0000 7f31039c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:44.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.713+0000 7f31039c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:21:44.717 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.713+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30fc0627b0 con 0x7f30fc13f710 2024-03-25T14:21:44.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.751+0000 7f30f1ffb640 1 -- 172.21.15.63:0/430765452 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f30ec050f50 con 0x7f30fc13f710 2024-03-25T14:21:44.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.752+0000 7f30f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:44.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.752+0000 7f30f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:44.755 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:44.755 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:44.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.754+0000 7f31039c9640 10 librados: watch_flush enter 2024-03-25T14:21:44.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.754+0000 7f31039c9640 10 librados: watch_flush exit 2024-03-25T14:21:44.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.754+0000 7f31039c9640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-25T14:21:44.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.754+0000 7f31039c9640 10 client.4301.objecter successfully canceled tick 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.754+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 >> v1:172.21.15.153:6800/1688255552 conn(0x7f30e006e0a0 legacy=0x7f30e0070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.754+0000 7f31039c9640 10 monclient: shutdown 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.755+0000 7f31039c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.755+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 >> v1:172.21.15.63:6790/0 conn(0x7f30fc13f710 legacy=0x7f30fc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.755+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 shutdown_connections 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.755+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 >> 172.21.15.63:0/430765452 conn(0x7f30fc0b7ea0 msgr2=0x7f30fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.755+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 shutdown_connections 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.755+0000 7f31039c9640 1 -- 172.21.15.63:0/430765452 wait complete. 2024-03-25T14:21:44.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:44.755+0000 7f31039c9640 1 librados: shutdown 2024-03-25T14:21:44.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:46.193 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:24.706 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.254 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:24.768 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.289 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:24.803 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.303 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:24.764 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.308 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:24.769 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.314 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:24.827 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.331 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:24.844 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.342 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:24.802 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:46.368 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:24.881 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:21:51.576 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:51.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.685+0000 7f6076b5c640 10 monclient: get_monmap_and_config 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.685+0000 7f6076b5c640 10 monclient: build_initial_monmap 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.685+0000 7f6076b5c640 10 monclient: monmap: 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:51.686821+0000 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:51.686821+0000 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:51.689 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.686+0000 7f6076b5c640 1 Processor -- start 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.686+0000 7f6076b5c640 1 -- start start 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.686+0000 7f6076b5c640 10 monclient: init 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 10 monclient(hunting): picked mon.noname-c con 0x7f60700a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 10 monclient(hunting): picked mon.noname-a con 0x7f607013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:51.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 10 monclient(hunting): picked mon.noname-b con 0x7f60701430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:51.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6070142520 con 0x7f607013f710 2024-03-25T14:21:51.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700b28a0 con 0x7f60700a98f0 2024-03-25T14:21:51.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700b2a30 con 0x7f60701430c0 2024-03-25T14:21:51.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:51.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.687+0000 7f6076b5c640 10 monclient(hunting): authenticate will time out at 1474.832165s 2024-03-25T14:21:51.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.698+0000 7f60750f9640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f60701430c0 0x7f607014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:49430/0 (socket says 172.21.15.63:49430) 2024-03-25T14:21:51.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.698+0000 7f60750f9640 1 -- 172.21.15.63:0/3070872846 learned_addr learned my addr 172.21.15.63:0/3070872846 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:51.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.706+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288953204 0 0) 0x7f60700b2a30 con 0x7f60701430c0 2024-03-25T14:21:51.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.706+0000 7f606f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:51.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.706+0000 7f606f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:51.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.706+0000 7f606f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:51.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.706+0000 7f606f7fe640 10 monclient(hunting): my global_id is 4304 2024-03-25T14:21:51.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.706+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6060003520 con 0x7f60701430c0 2024-03-25T14:21:51.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.720+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585493469 0 0) 0x7f6070142520 con 0x7f607013f710 2024-03-25T14:21:51.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.720+0000 7f606f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:51.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.720+0000 7f606f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:51.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.720+0000 7f606f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:51.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.720+0000 7f606f7fe640 10 monclient(hunting): my global_id is 4305 2024-03-25T14:21:51.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.720+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60700b2a30 con 0x7f607013f710 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 999241890 0 0) 0x7f6060003520 con 0x7f60701430c0 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 >> v1:172.21.15.153:6789/0 conn(0x7f60700a98f0 legacy=0x7f60700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 >> v1:172.21.15.63:6789/0 conn(0x7f607013f710 legacy=0x7f60701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 1 monclient: found mon.noname-b 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60700b44c0 con 0x7f60701430c0 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:21:51.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f6076b5c640 5 monclient: authenticate success, global_id 4304 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f6076b5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f60680019d0 con 0x7f60701430c0 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f606f7fe640 10 monclient: dump: 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:51.726 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:51.727 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:51.727 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:51.727 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:51.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.721+0000 7f6076b5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:51.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6068001c20 con 0x7f60701430c0 2024-03-25T14:21:51.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f606f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:51.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f606f7fe640 1 -- 172.21.15.63:0/3070872846 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6068001fb0 con 0x7f60701430c0 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f606f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f606f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f606f7fe640 10 monclient: dump: 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:51.751 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:51.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f6076b5c640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:51.752 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.747+0000 7f6076b5c640 4 set_mon_vals no callback set 2024-03-25T14:21:51.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.748+0000 7f6076b5c640 10 monclient: shutdown 2024-03-25T14:21:51.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.748+0000 7f6076b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:51.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.748+0000 7f6076b5c640 1 -- 172.21.15.63:0/3070872846 >> v1:172.21.15.63:6790/0 conn(0x7f60701430c0 legacy=0x7f607014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:51.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.748+0000 7f6076b5c640 1 -- 172.21.15.63:0/3070872846 shutdown_connections 2024-03-25T14:21:51.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.748+0000 7f6076b5c640 1 -- 172.21.15.63:0/3070872846 >> 172.21.15.63:0/3070872846 conn(0x7f60700b7ea0 msgr2=0x7f60700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:51.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.748+0000 7f6076b5c640 1 -- 172.21.15.63:0/3070872846 shutdown_connections 2024-03-25T14:21:51.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.748+0000 7f6076b5c640 1 -- 172.21.15.63:0/3070872846 wait complete. 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.749+0000 7f6076b5c640 10 monclient: build_initial_monmap 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.749+0000 7f6076b5c640 10 monclient: monmap: 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:51.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.750+0000 7f6076b5c640 1 librados: starting msgr at 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.750+0000 7f6076b5c640 1 librados: starting objecter 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.750+0000 7f6076b5c640 1 Processor -- start 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.750+0000 7f6076b5c640 1 -- start start 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.750+0000 7f6076b5c640 1 librados: setting wanted keys 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.750+0000 7f6076b5c640 1 librados: calling monclient init 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.750+0000 7f6076b5c640 10 monclient: init 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:51.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 10 monclient(hunting): picked mon.noname-a con 0x7f60700a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 10 monclient(hunting): picked mon.noname-c con 0x7f607013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 10 monclient(hunting): picked mon.noname-b con 0x7f60701430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700704c0 con 0x7f60700a98f0 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607006f2c0 con 0x7f60701430c0 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607006f450 con 0x7f607013f710 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:51.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.751+0000 7f6076b5c640 10 monclient(hunting): authenticate will time out at 1474.896013s 2024-03-25T14:21:51.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.763+0000 7f60748f8640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f60700a98f0 0x7f607006ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49244/0 (socket says 172.21.15.63:49244) 2024-03-25T14:21:51.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.763+0000 7f60748f8640 1 -- 172.21.15.63:0/4150165476 learned_addr learned my addr 172.21.15.63:0/4150165476 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:51.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.766+0000 7f606d7fa640 10 client.?.objecter ms_handle_connect 0x7f60701430c0 2024-03-25T14:21:51.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.766+0000 7f606d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:51.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.766+0000 7f606d7fa640 10 client.?.objecter ms_handle_connect 0x7f60700a98f0 2024-03-25T14:21:51.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.766+0000 7f606d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:51.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.771+0000 7f606d7fa640 10 client.?.objecter ms_handle_connect 0x7f607013f710 2024-03-25T14:21:51.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.771+0000 7f606d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.772+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376370770 0 0) 0x7f60700704c0 con 0x7f60700a98f0 2024-03-25T14:21:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.772+0000 7f606d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.772+0000 7f606d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.772+0000 7f606d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.772+0000 7f606d7fa640 10 monclient(hunting): my global_id is 4308 2024-03-25T14:21:51.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.772+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6054004eb0 con 0x7f60700a98f0 2024-03-25T14:21:51.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.777+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160894086 0 0) 0x7f607006f2c0 con 0x7f60701430c0 2024-03-25T14:21:51.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.777+0000 7f606d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:51.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.777+0000 7f606d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:51.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.777+0000 7f606d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:51.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.777+0000 7f606d7fa640 10 monclient(hunting): my global_id is 4321 2024-03-25T14:21:51.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.777+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60700704c0 con 0x7f60701430c0 2024-03-25T14:21:51.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.782+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204964682 0 0) 0x7f607006f450 con 0x7f607013f710 2024-03-25T14:21:51.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.782+0000 7f606d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:51.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.782+0000 7f606d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:51.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.782+0000 7f606d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:51.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.782+0000 7f606d7fa640 10 monclient(hunting): my global_id is 4307 2024-03-25T14:21:51.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.782+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607006f2c0 con 0x7f607013f710 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.786+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1219185902 0 0) 0x7f6054004eb0 con 0x7f60700a98f0 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.786+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f607006f450 con 0x7f60700a98f0 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.786+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f60640019d0 con 0x7f60700a98f0 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.786+0000 7f606d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.786+0000 7f606d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.786+0000 7f606d7fa640 10 monclient(hunting): dump: 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.790 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:51.791 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:51.791 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:51.791 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:51.791 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:51.791 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:51.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.786+0000 7f606d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:51.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.794+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1255716419 0 0) 0x7f60700704c0 con 0x7f60701430c0 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.794+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6054004eb0 con 0x7f60701430c0 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.794+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6068001f30 con 0x7f60701430c0 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.794+0000 7f606d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.795+0000 7f606d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.795+0000 7f606d7fa640 10 monclient(hunting): dump: 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:51.798 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:51.799 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:51.799 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:51.799 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:51.799 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:51.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3931824709 0 0) 0x7f6054004eb0 con 0x7f60701430c0 2024-03-25T14:21:51.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 >> v1:172.21.15.63:6790/0 conn(0x7f607013f710 legacy=0x7f60701dbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:51.809 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 >> v1:172.21.15.63:6789/0 conn(0x7f60700a98f0 legacy=0x7f607006ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 1 monclient: found mon.b 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6070070950 con 0x7f60701430c0 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f606d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 5 monclient: authenticate success, global_id 4321 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 10 monclient: _renew_subs 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60701dc6d0 con 0x7f60701430c0 2024-03-25T14:21:51.810 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:51.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 10 monclient: _renew_subs 2024-03-25T14:21:51.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:51.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f607006e080 con 0x7f60701430c0 2024-03-25T14:21:51.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.806+0000 7f6076b5c640 1 librados: init done 2024-03-25T14:21:51.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.807+0000 7f6076b5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:51.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.807+0000 7f6076b5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:51.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.807+0000 7f6076b5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:51.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:51.807+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6070070b40 con 0x7f60701430c0 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.092+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6068002090 con 0x7f60701430c0 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.092+0000 7f606d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.092+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6068004fb0 con 0x7f60701430c0 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.092+0000 7f606d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.092+0000 7f606d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.092+0000 7f606d7fa640 10 monclient: dump: 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:52.095 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.092+0000 7f606effd640 4 set_mon_vals no callback set 2024-03-25T14:21:52.096 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.093+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f606801af80 con 0x7f60701430c0 2024-03-25T14:21:52.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.093+0000 7f60748f8640 10 monclient: get_auth_request con 0x7f605406dfb0 auth_method 0 2024-03-25T14:21:52.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.094+0000 7f606d7fa640 10 client.4321.objecter ms_handle_connect 0x7f605406dfb0 2024-03-25T14:21:52.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f6068054b20 con 0x7f60701430c0 2024-03-25T14:21:52.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 10 client.4321.objecter ms_dispatch 0x7f6070000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:52.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:52.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:52.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T14:21:52.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6068050460 con 0x7f60701430c0 2024-03-25T14:21:52.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:52.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.099+0000 7f606d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:52.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.363+0000 7f6076b5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:52.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.363+0000 7f6076b5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:52.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.363+0000 7f6076b5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:52.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.363+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60700627b0 con 0x7f60701430c0 2024-03-25T14:21:52.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.393+0000 7f606d7fa640 1 -- 172.21.15.63:0/4150165476 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f6068052560 con 0x7f60701430c0 2024-03-25T14:21:52.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.393+0000 7f606d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:52.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.393+0000 7f606d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:52.397 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:52.397 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 10 librados: watch_flush enter 2024-03-25T14:21:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 10 librados: watch_flush exit 2024-03-25T14:21:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T14:21:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 10 client.4321.objecter successfully canceled tick 2024-03-25T14:21:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 >> v1:172.21.15.153:6800/1688255552 conn(0x7f605406dfb0 legacy=0x7f6054070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 10 monclient: shutdown 2024-03-25T14:21:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 >> v1:172.21.15.153:6789/0 conn(0x7f60701430c0 legacy=0x7f60701e6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 shutdown_connections 2024-03-25T14:21:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 >> 172.21.15.63:0/4150165476 conn(0x7f60700b7ea0 msgr2=0x7f6070074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 shutdown_connections 2024-03-25T14:21:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 1 -- 172.21.15.63:0/4150165476 wait complete. 2024-03-25T14:21:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:52.396+0000 7f6076b5c640 1 librados: shutdown 2024-03-25T14:21:52.411 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:21:59.219 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:21:59.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.313+0000 7f2b7ed9d640 10 monclient: get_monmap_and_config 2024-03-25T14:21:59.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.313+0000 7f2b7ed9d640 10 monclient: build_initial_monmap 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.313+0000 7f2b7ed9d640 10 monclient: monmap: 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:21:59.314581+0000 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:21:59.314581+0000 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:21:59.317 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:59.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 1 Processor -- start 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 1 -- start start 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 10 monclient: init 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2b780a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b7813f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.314+0000 7f2b7ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b781430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.315+0000 7f2b7ed9d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b78142520 con 0x7f2b780a98f0 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.315+0000 7f2b7ed9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b780b28a0 con 0x7f2b781430c0 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.315+0000 7f2b7ed9d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b780b2a30 con 0x7f2b7813f710 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.315+0000 7f2b7ed9d640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:59.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.315+0000 7f2b7ed9d640 10 monclient(hunting): authenticate will time out at 1482.459581s 2024-03-25T14:21:59.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.336+0000 7f2b7cb39640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2b780a98f0 0x7f2b780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:53666/0 (socket says 172.21.15.63:53666) 2024-03-25T14:21:59.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.336+0000 7f2b7cb39640 1 -- 172.21.15.63:0/1970066005 learned_addr learned my addr 172.21.15.63:0/1970066005 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:59.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.359+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934742630 0 0) 0x7f2b78142520 con 0x7f2b780a98f0 2024-03-25T14:21:59.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.359+0000 7f2b777fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:59.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.359+0000 7f2b777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:59.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.359+0000 7f2b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:59.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.359+0000 7f2b777fe640 10 monclient(hunting): my global_id is 4311 2024-03-25T14:21:59.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.359+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b68003520 con 0x7f2b780a98f0 2024-03-25T14:21:59.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.363+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381466056 0 0) 0x7f2b780b28a0 con 0x7f2b781430c0 2024-03-25T14:21:59.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.363+0000 7f2b777fe640 10 monclient(hunting): _negotiate 2024-03-25T14:21:59.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.363+0000 7f2b777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:59.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.363+0000 7f2b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:59.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.363+0000 7f2b777fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T14:21:59.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.363+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b78142520 con 0x7f2b781430c0 2024-03-25T14:21:59.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3996892544 0 0) 0x7f2b68003520 con 0x7f2b780a98f0 2024-03-25T14:21:59.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 >> v1:172.21.15.63:6790/0 conn(0x7f2b7813f710 legacy=0x7f2b781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 >> v1:172.21.15.153:6789/0 conn(0x7f2b781430c0 legacy=0x7f2b7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 10 monclient: _finish_hunting 0 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 1 monclient: found mon.noname-a 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b780b44c0 con 0x7f2b780a98f0 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 10 monclient: _finish_auth 0 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 10 monclient: _check_auth_tickets 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2b6c001690 con 0x7f2b780a98f0 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b777fe640 10 monclient: dump: 2024-03-25T14:21:59.371 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b7ed9d640 5 monclient: authenticate success, global_id 4311 2024-03-25T14:21:59.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.367+0000 7f2b7ed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b6c0017f0 con 0x7f2b780a98f0 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b777fe640 1 -- 172.21.15.63:0/1970066005 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2b6c001c00 con 0x7f2b780a98f0 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b777fe640 10 monclient: dump: 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:59.390 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b7ed9d640 10 monclient: get_monmap_and_config success 2024-03-25T14:21:59.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.387+0000 7f2b7ed9d640 4 set_mon_vals no callback set 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.388+0000 7f2b7ed9d640 10 monclient: shutdown 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.388+0000 7f2b7ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.388+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/1970066005 >> v1:172.21.15.63:6789/0 conn(0x7f2b780a98f0 legacy=0x7f2b780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.388+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/1970066005 shutdown_connections 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.388+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/1970066005 >> 172.21.15.63:0/1970066005 conn(0x7f2b780b7ea0 msgr2=0x7f2b780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.388+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/1970066005 shutdown_connections 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.388+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/1970066005 wait complete. 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 10 monclient: build_initial_monmap 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 10 monclient: monmap: 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:21:59.393 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 1 librados: starting msgr at 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 1 librados: starting objecter 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 1 Processor -- start 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 1 -- start start 2024-03-25T14:21:59.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 1 librados: setting wanted keys 2024-03-25T14:21:59.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 1 librados: calling monclient init 2024-03-25T14:21:59.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.389+0000 7f2b7ed9d640 10 monclient: init 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 10 monclient: _reopen_session rank -1 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f2b780a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b7813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b781430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7813cc80 con 0x7f2b780a98f0 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7813b180 con 0x7f2b7813f710 2024-03-25T14:21:59.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7813b460 con 0x7f2b781430c0 2024-03-25T14:21:59.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 10 monclient(hunting): _renew_subs 2024-03-25T14:21:59.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.390+0000 7f2b7ed9d640 10 monclient(hunting): authenticate will time out at 1482.535510s 2024-03-25T14:21:59.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.399+0000 7f2b7d33a640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f2b781430c0 0x7f2b781e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:53988/0 (socket says 172.21.15.63:53988) 2024-03-25T14:21:59.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.399+0000 7f2b7d33a640 1 -- 172.21.15.63:0/2173189457 learned_addr learned my addr 172.21.15.63:0/2173189457 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:21:59.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.400+0000 7f2b757fa640 10 client.?.objecter ms_handle_connect 0x7f2b781430c0 2024-03-25T14:21:59.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.400+0000 7f2b757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:59.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.400+0000 7f2b757fa640 10 client.?.objecter ms_handle_connect 0x7f2b7813f710 2024-03-25T14:21:59.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.400+0000 7f2b757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:59.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.405+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823987578 0 0) 0x7f2b7813b180 con 0x7f2b7813f710 2024-03-25T14:21:59.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.405+0000 7f2b757fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:59.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.405+0000 7f2b757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:59.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.405+0000 7f2b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:59.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.405+0000 7f2b757fa640 10 monclient(hunting): my global_id is 4327 2024-03-25T14:21:59.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.405+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5c004eb0 con 0x7f2b7813f710 2024-03-25T14:21:59.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 10 client.?.objecter ms_handle_connect 0x7f2b780a98f0 2024-03-25T14:21:59.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:21:59.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459182002 0 0) 0x7f2b7813b460 con 0x7f2b781430c0 2024-03-25T14:21:59.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 10 monclient(hunting): _negotiate 2024-03-25T14:21:59.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:21:59.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:21:59.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 10 monclient(hunting): my global_id is 4310 2024-03-25T14:21:59.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.410+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b7813b180 con 0x7f2b781430c0 2024-03-25T14:21:59.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.416+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1011500282 0 0) 0x7f2b5c004eb0 con 0x7f2b7813f710 2024-03-25T14:21:59.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.416+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b7813b460 con 0x7f2b7813f710 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.416+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2b64001870 con 0x7f2b7813f710 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.416+0000 7f2b757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.416+0000 7f2b757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.416+0000 7f2b757fa640 10 monclient(hunting): dump: 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:59.420 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:59.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.416+0000 7f2b757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:21:59.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.426+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313539281 0 0) 0x7f2b7813b180 con 0x7f2b781430c0 2024-03-25T14:21:59.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.426+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b5c004eb0 con 0x7f2b781430c0 2024-03-25T14:21:59.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.426+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2b70001c50 con 0x7f2b781430c0 2024-03-25T14:21:59.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.426+0000 7f2b757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:59.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.426+0000 7f2b757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:21:59.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.426+0000 7f2b757fa640 10 monclient(hunting): dump: 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:59.430 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 805434427 0 0) 0x7f2b7813b460 con 0x7f2b7813f710 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 >> v1:172.21.15.63:6790/0 conn(0x7f2b781430c0 legacy=0x7f2b781e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 >> v1:172.21.15.63:6789/0 conn(0x7f2b780a98f0 legacy=0x7f2b78139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 10 monclient: _finish_hunting 0 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 1 monclient: found mon.b 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:59.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7813b740 con 0x7f2b7813f710 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 10 monclient: _finish_auth 0 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 10 monclient: _check_auth_tickets 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b7ed9d640 5 monclient: authenticate success, global_id 4327 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b7ed9d640 10 monclient: _renew_subs 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b7ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b780b7210 con 0x7f2b7813f710 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b7ed9d640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.428+0000 7f2b7ed9d640 10 monclient: _renew_subs 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.429+0000 7f2b7ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.429+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b78139940 con 0x7f2b7813f710 2024-03-25T14:21:59.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.429+0000 7f2b7ed9d640 1 librados: init done 2024-03-25T14:21:59.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.430+0000 7f2b7ed9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:59.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.430+0000 7f2b7ed9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:59.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.430+0000 7f2b7ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:59.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.430+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7814f320 con 0x7f2b7813f710 2024-03-25T14:21:59.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.703+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b640035f0 con 0x7f2b7813f710 2024-03-25T14:21:59.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.703+0000 7f2b757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:21:59.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.703+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2b64001d80 con 0x7f2b7813f710 2024-03-25T14:21:59.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.703+0000 7f2b757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.703+0000 7f2b757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.703+0000 7f2b757fa640 10 monclient: dump: 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:21:59.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.703+0000 7f2b76ffd640 4 set_mon_vals no callback set 2024-03-25T14:21:59.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.704+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f2b640197f0 con 0x7f2b7813f710 2024-03-25T14:21:59.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.704+0000 7f2b7cb39640 10 monclient: get_auth_request con 0x7f2b5c06ded0 auth_method 0 2024-03-25T14:21:59.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.705+0000 7f2b757fa640 10 client.4327.objecter ms_handle_connect 0x7f2b5c06ded0 2024-03-25T14:21:59.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.709+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f2b64053570 con 0x7f2b7813f710 2024-03-25T14:21:59.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.709+0000 7f2b757fa640 10 client.4327.objecter ms_dispatch 0x7f2b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:21:59.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.709+0000 7f2b757fa640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:21:59.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.709+0000 7f2b757fa640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:21:59.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.709+0000 7f2b757fa640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T14:21:59.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.710+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b6404eeb0 con 0x7f2b7813f710 2024-03-25T14:21:59.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.710+0000 7f2b757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:21:59.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.710+0000 7f2b757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:21:59.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.957+0000 7f2b7ed9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:59.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.957+0000 7f2b7ed9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:59.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.957+0000 7f2b7ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:21:59.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.957+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b78074cc0 con 0x7f2b7813f710 2024-03-25T14:21:59.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.985+0000 7f2b757fa640 1 -- 172.21.15.63:0/2173189457 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f2b64050fb0 con 0x7f2b7813f710 2024-03-25T14:21:59.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.985+0000 7f2b757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:21:59.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.985+0000 7f2b757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:21:59.989 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:21:59.989 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 10 librados: watch_flush enter 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 10 librados: watch_flush exit 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 10 client.4327.objecter successfully canceled tick 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 >> v1:172.21.15.153:6800/1688255552 conn(0x7f2b5c06ded0 legacy=0x7f2b5c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 10 monclient: shutdown 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:21:59.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 >> v1:172.21.15.153:6789/0 conn(0x7f2b7813f710 legacy=0x7f2b781df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:21:59.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 shutdown_connections 2024-03-25T14:21:59.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.987+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 >> 172.21.15.63:0/2173189457 conn(0x7f2b780b7ea0 msgr2=0x7f2b78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:21:59.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.988+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 shutdown_connections 2024-03-25T14:21:59.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.988+0000 7f2b7ed9d640 1 -- 172.21.15.63:0/2173189457 wait complete. 2024-03-25T14:21:59.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:21:59.988+0000 7f2b7ed9d640 1 librados: shutdown 2024-03-25T14:22:00.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:22:06.184 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:44.697 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.191 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:44.705 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.223 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:02:44.737 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.233 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:44.693 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.238 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:44.751 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.241 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:44.702 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.250 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:44.764 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.283 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:44.744 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.297 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:02:44.811 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:06.811 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:22:06.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.923+0000 7ff129f5d640 10 monclient: get_monmap_and_config 2024-03-25T14:22:06.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.923+0000 7ff129f5d640 10 monclient: build_initial_monmap 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.923+0000 7ff129f5d640 10 monclient: monmap: 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:22:06.924262+0000 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:22:06.924262+0000 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:22:06.927 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.924+0000 7ff129f5d640 1 Processor -- start 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.924+0000 7ff129f5d640 1 -- start start 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.924+0000 7ff129f5d640 10 monclient: init 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff1240a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff12413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff1241430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:06.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff124142520 con 0x7ff1241430c0 2024-03-25T14:22:06.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1240b28a0 con 0x7ff1240a98f0 2024-03-25T14:22:06.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1240b2a30 con 0x7ff12413f710 2024-03-25T14:22:06.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:06.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.925+0000 7ff129f5d640 10 monclient(hunting): authenticate will time out at 1490.070475s 2024-03-25T14:22:06.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.935+0000 7ff1237fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff1240a98f0 0x7ff1240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:51252/0 (socket says 172.21.15.63:51252) 2024-03-25T14:22:06.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.935+0000 7ff1237fe640 1 -- 172.21.15.63:0/4055650823 learned_addr learned my addr 172.21.15.63:0/4055650823 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:06.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.942+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030219288 0 0) 0x7ff1240b28a0 con 0x7ff1240a98f0 2024-03-25T14:22:06.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.942+0000 7ff1227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:06.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.942+0000 7ff1227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:06.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.942+0000 7ff1227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:06.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.942+0000 7ff1227fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T14:22:06.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.942+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff10c003520 con 0x7ff1240a98f0 2024-03-25T14:22:06.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.949+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187374096 0 0) 0x7ff124142520 con 0x7ff1241430c0 2024-03-25T14:22:06.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.949+0000 7ff1227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:06.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.949+0000 7ff1227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.949+0000 7ff1227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.949+0000 7ff1227fc640 10 monclient(hunting): my global_id is 4317 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.949+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1240b28a0 con 0x7ff1241430c0 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003044198 0 0) 0x7ff10c003520 con 0x7ff1240a98f0 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 >> v1:172.21.15.63:6790/0 conn(0x7ff12413f710 legacy=0x7ff1241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 >> v1:172.21.15.63:6789/0 conn(0x7ff1241430c0 legacy=0x7ff12414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 10 monclient: _finish_hunting 0 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 1 monclient: found mon.noname-c 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:22:06.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240b44c0 con 0x7ff1240a98f0 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 10 monclient: _finish_auth 0 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 10 monclient: _check_auth_tickets 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff114001690 con 0x7ff1240a98f0 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff1227fc640 10 monclient: dump: 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:06.957 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:06.958 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:06.958 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:06.958 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:06.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff129f5d640 5 monclient: authenticate success, global_id 4330 2024-03-25T14:22:06.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.950+0000 7ff129f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:06.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1140017f0 con 0x7ff1240a98f0 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff1227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff1227fc640 1 -- 172.21.15.63:0/4055650823 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff114001c00 con 0x7ff1240a98f0 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff1227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff1227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff1227fc640 10 monclient: dump: 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:06.973 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:06.974 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:06.974 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:06.974 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:06.974 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:06.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff129f5d640 10 monclient: get_monmap_and_config success 2024-03-25T14:22:06.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.969+0000 7ff129f5d640 4 set_mon_vals no callback set 2024-03-25T14:22:06.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.970+0000 7ff129f5d640 10 monclient: shutdown 2024-03-25T14:22:06.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.970+0000 7ff129f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:06.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.970+0000 7ff129f5d640 1 -- 172.21.15.63:0/4055650823 >> v1:172.21.15.153:6789/0 conn(0x7ff1240a98f0 legacy=0x7ff1240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:06.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.970+0000 7ff129f5d640 1 -- 172.21.15.63:0/4055650823 shutdown_connections 2024-03-25T14:22:06.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.970+0000 7ff129f5d640 1 -- 172.21.15.63:0/4055650823 >> 172.21.15.63:0/4055650823 conn(0x7ff1240b7ea0 msgr2=0x7ff1240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:06.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.970+0000 7ff129f5d640 1 -- 172.21.15.63:0/4055650823 shutdown_connections 2024-03-25T14:22:06.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.970+0000 7ff129f5d640 1 -- 172.21.15.63:0/4055650823 wait complete. 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.971+0000 7ff129f5d640 10 monclient: build_initial_monmap 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.971+0000 7ff129f5d640 10 monclient: monmap: 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:22:06.977 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.971+0000 7ff129f5d640 1 librados: starting msgr at 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.971+0000 7ff129f5d640 1 librados: starting objecter 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.971+0000 7ff129f5d640 1 Processor -- start 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 1 -- start start 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 1 librados: setting wanted keys 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 1 librados: calling monclient init 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient: init 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff1240a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff12413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff1241430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:06.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff124138250 con 0x7ff1240a98f0 2024-03-25T14:22:06.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff124137d50 con 0x7ff1241430c0 2024-03-25T14:22:06.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff124137ee0 con 0x7ff12413f710 2024-03-25T14:22:06.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:06.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.972+0000 7ff129f5d640 10 monclient(hunting): authenticate will time out at 1490.117317s 2024-03-25T14:22:06.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.984+0000 7ff123fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff1241430c0 0x7ff1241d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:51274/0 (socket says 172.21.15.63:51274) 2024-03-25T14:22:06.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.984+0000 7ff123fff640 1 -- 172.21.15.63:0/3771416826 learned_addr learned my addr 172.21.15.63:0/3771416826 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:06.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.985+0000 7ff10bfff640 10 client.?.objecter ms_handle_connect 0x7ff1241430c0 2024-03-25T14:22:06.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.985+0000 7ff10bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:06.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.988+0000 7ff10bfff640 10 client.?.objecter ms_handle_connect 0x7ff12413f710 2024-03-25T14:22:06.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.988+0000 7ff10bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:06.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.992+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897445095 0 0) 0x7ff124137d50 con 0x7ff1241430c0 2024-03-25T14:22:06.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.992+0000 7ff10bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:22:06.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.992+0000 7ff10bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:06.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.992+0000 7ff10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:06.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.992+0000 7ff10bfff640 10 monclient(hunting): my global_id is 4333 2024-03-25T14:22:06.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.992+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff110004eb0 con 0x7ff1241430c0 2024-03-25T14:22:06.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.995+0000 7ff10bfff640 10 client.?.objecter ms_handle_connect 0x7ff1240a98f0 2024-03-25T14:22:06.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:06.995+0000 7ff10bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:07.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666409943 0 0) 0x7ff110004eb0 con 0x7ff1241430c0 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff124137d50 con 0x7ff1241430c0 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff11c001c90 con 0x7ff1241430c0 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): dump: 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:07.009 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939669904 0 0) 0x7ff124137ee0 con 0x7ff12413f710 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 10 monclient(hunting): my global_id is 4316 2024-03-25T14:22:07.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.005+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff110004eb0 con 0x7ff12413f710 2024-03-25T14:22:07.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.006+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674972669 0 0) 0x7ff124138250 con 0x7ff1240a98f0 2024-03-25T14:22:07.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.006+0000 7ff10bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:22:07.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.006+0000 7ff10bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:07.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.006+0000 7ff10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:07.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.006+0000 7ff10bfff640 10 monclient(hunting): my global_id is 4320 2024-03-25T14:22:07.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.006+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff124137ee0 con 0x7ff1240a98f0 2024-03-25T14:22:07.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3545454366 0 0) 0x7ff124137d50 con 0x7ff1241430c0 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 >> v1:172.21.15.63:6790/0 conn(0x7ff12413f710 legacy=0x7ff1241d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 >> v1:172.21.15.63:6789/0 conn(0x7ff1240a98f0 legacy=0x7ff124137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 1 monclient: found mon.b 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240b7730 con 0x7ff1241430c0 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 10 monclient: _finish_auth 0 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff10bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 5 monclient: authenticate success, global_id 4333 2024-03-25T14:22:07.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 10 monclient: _renew_subs 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1240b7920 con 0x7ff1241430c0 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 10 monclient: _renew_subs 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1241e33d0 con 0x7ff1241430c0 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.011+0000 7ff129f5d640 1 librados: init done 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.012+0000 7ff129f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.012+0000 7ff129f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.012+0000 7ff129f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:07.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.012+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1240627b0 con 0x7ff1241430c0 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.291+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff11c003990 con 0x7ff1241430c0 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.291+0000 7ff10bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.291+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff11c003da0 con 0x7ff1241430c0 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.291+0000 7ff10bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.292+0000 7ff10bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.292+0000 7ff10bfff640 10 monclient: dump: 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:07.295 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:07.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.292+0000 7ff121ffb640 4 set_mon_vals no callback set 2024-03-25T14:22:07.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.292+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7ff11c003fc0 con 0x7ff1241430c0 2024-03-25T14:22:07.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.293+0000 7ff1237fe640 10 monclient: get_auth_request con 0x7ff11006de80 auth_method 0 2024-03-25T14:22:07.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.293+0000 7ff10bfff640 10 client.4333.objecter ms_handle_connect 0x7ff11006de80 2024-03-25T14:22:07.300 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.297+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7ff11c054af0 con 0x7ff1241430c0 2024-03-25T14:22:07.300 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.297+0000 7ff10bfff640 10 client.4333.objecter ms_dispatch 0x7ff124000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:22:07.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.297+0000 7ff10bfff640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:22:07.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.297+0000 7ff10bfff640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:22:07.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.297+0000 7ff10bfff640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T14:22:07.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.298+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff11c050600 con 0x7ff1241430c0 2024-03-25T14:22:07.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.298+0000 7ff10bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:07.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.298+0000 7ff10bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:22:07.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.523+0000 7ff129f5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:07.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.523+0000 7ff129f5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:07.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.523+0000 7ff129f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:07.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.523+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff124138400 con 0x7ff1241430c0 2024-03-25T14:22:07.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.601+0000 7ff10bfff640 1 -- 172.21.15.63:0/3771416826 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7ff124138400 con 0x7ff1241430c0 2024-03-25T14:22:07.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.601+0000 7ff10bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:07.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.601+0000 7ff10bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:22:07.605 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:22:07.605 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 10 librados: watch_flush enter 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 10 librados: watch_flush exit 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 10 client.4333.objecter successfully canceled tick 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 >> v1:172.21.15.153:6800/1688255552 conn(0x7ff11006de80 legacy=0x7ff110070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 10 monclient: shutdown 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 >> v1:172.21.15.153:6789/0 conn(0x7ff1241430c0 legacy=0x7ff1241d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 shutdown_connections 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 >> 172.21.15.63:0/3771416826 conn(0x7ff1240b7ea0 msgr2=0x7ff124074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 shutdown_connections 2024-03-25T14:22:07.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 1 -- 172.21.15.63:0/3771416826 wait complete. 2024-03-25T14:22:07.609 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:07.604+0000 7ff129f5d640 1 librados: shutdown 2024-03-25T14:22:07.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:22:14.430 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:22:14.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.545+0000 7f9ff3bf4640 10 monclient: get_monmap_and_config 2024-03-25T14:22:14.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.545+0000 7f9ff3bf4640 10 monclient: build_initial_monmap 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.545+0000 7f9ff3bf4640 10 monclient: monmap: 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:22:14.546626+0000 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:22:14.546626+0000 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:22:14.549 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.546+0000 7f9ff3bf4640 1 Processor -- start 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.546+0000 7f9ff3bf4640 1 -- start start 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.546+0000 7f9ff3bf4640 10 monclient: init 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.547+0000 7f9ff3bf4640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.547+0000 7f9ff3bf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.547+0000 7f9ff3bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec0b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.547+0000 7f9ff3bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec0b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:14.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.547+0000 7f9ff3bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec14be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:14.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.547+0000 7f9ff3bf4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0af940 con 0x7f9fec0b4d60 2024-03-25T14:22:14.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.548+0000 7f9ff3bf4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0b0a70 con 0x7f9fec0b1730 2024-03-25T14:22:14.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.548+0000 7f9ff3bf4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0b0d50 con 0x7f9fec14be90 2024-03-25T14:22:14.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.548+0000 7f9ff3bf4640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:14.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.548+0000 7f9ff3bf4640 10 monclient(hunting): authenticate will time out at 1497.692774s 2024-03-25T14:22:14.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.554+0000 7f9ff1990640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f9fec0b4d60 0x7f9fec13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47570/0 (socket says 172.21.15.63:47570) 2024-03-25T14:22:14.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.554+0000 7f9ff1990640 1 -- 172.21.15.63:0/508310588 learned_addr learned my addr 172.21.15.63:0/508310588 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575758473 0 0) 0x7f9fec0b0a70 con 0x7f9fec0b1730 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): _negotiate 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): my global_id is 4336 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd8003520 con 0x7f9fec0b1730 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260591127 0 0) 0x7f9fec0af940 con 0x7f9fec0b4d60 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): _negotiate 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 10 monclient(hunting): my global_id is 4323 2024-03-25T14:22:14.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.574+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec0b0a70 con 0x7f9fec0b4d60 2024-03-25T14:22:14.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.584+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406361550 0 0) 0x7f9fec0b0d50 con 0x7f9fec14be90 2024-03-25T14:22:14.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.584+0000 7f9ff098e640 10 monclient(hunting): _negotiate 2024-03-25T14:22:14.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.584+0000 7f9ff098e640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:14.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.584+0000 7f9ff098e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:14.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.584+0000 7f9ff098e640 10 monclient(hunting): my global_id is 4319 2024-03-25T14:22:14.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.584+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec0af940 con 0x7f9fec14be90 2024-03-25T14:22:14.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632160273 0 0) 0x7f9fec0b0a70 con 0x7f9fec0b4d60 2024-03-25T14:22:14.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 >> v1:172.21.15.63:6790/0 conn(0x7f9fec14be90 legacy=0x7f9fec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:14.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 >> v1:172.21.15.153:6789/0 conn(0x7f9fec0b1730 legacy=0x7f9fec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 10 monclient: _finish_hunting 0 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 1 monclient: found mon.noname-a 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec0b1030 con 0x7f9fec0b4d60 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 10 monclient: _finish_auth 0 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 10 monclient: _check_auth_tickets 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9fe8001670 con 0x7f9fec0b4d60 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:22:14.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff098e640 10 monclient: dump: 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:14.591 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:14.592 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:14.592 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:14.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff3bf4640 5 monclient: authenticate success, global_id 4323 2024-03-25T14:22:14.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.586+0000 7f9ff3bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe80017d0 con 0x7f9fec0b4d60 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff098e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff098e640 1 -- 172.21.15.63:0/508310588 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9fe8001be0 con 0x7f9fec0b4d60 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff098e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff098e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff098e640 10 monclient: dump: 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.624 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff3bf4640 10 monclient: get_monmap_and_config success 2024-03-25T14:22:14.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.621+0000 7f9ff3bf4640 4 set_mon_vals no callback set 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.622+0000 7f9ff3bf4640 10 monclient: shutdown 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.622+0000 7f9ff3bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.622+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/508310588 >> v1:172.21.15.63:6789/0 conn(0x7f9fec0b4d60 legacy=0x7f9fec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.622+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/508310588 shutdown_connections 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.622+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/508310588 >> 172.21.15.63:0/508310588 conn(0x7f9fec0617a0 msgr2=0x7f9fec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.622+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/508310588 shutdown_connections 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.622+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/508310588 wait complete. 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.623+0000 7f9ff3bf4640 10 monclient: build_initial_monmap 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.623+0000 7f9ff3bf4640 10 monclient: monmap: 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:14.627 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.623+0000 7f9ff3bf4640 1 librados: starting msgr at 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.623+0000 7f9ff3bf4640 1 librados: starting objecter 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.623+0000 7f9ff3bf4640 1 Processor -- start 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 1 -- start start 2024-03-25T14:22:14.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 1 librados: setting wanted keys 2024-03-25T14:22:14.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 1 librados: calling monclient init 2024-03-25T14:22:14.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient: init 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec0b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec0b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec14be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec143d40 con 0x7f9fec0b4d60 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec140f00 con 0x7f9fec0b1730 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec138460 con 0x7f9fec14be90 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:14.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.624+0000 7f9ff3bf4640 10 monclient(hunting): authenticate will time out at 1497.769260s 2024-03-25T14:22:14.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.629+0000 7f9ff2191640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f9fec14be90 0x7f9fec1d7d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:60456/0 (socket says 172.21.15.63:60456) 2024-03-25T14:22:14.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.629+0000 7f9ff2191640 1 -- 172.21.15.63:0/187378389 learned_addr learned my addr 172.21.15.63:0/187378389 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:14.636 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.633+0000 7f9fe27fc640 10 client.?.objecter ms_handle_connect 0x7f9fec14be90 2024-03-25T14:22:14.636 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.633+0000 7f9fe27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:14.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.638+0000 7f9fe27fc640 10 client.?.objecter ms_handle_connect 0x7f9fec0b1730 2024-03-25T14:22:14.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.638+0000 7f9fe27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:14.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.639+0000 7f9fe27fc640 10 client.?.objecter ms_handle_connect 0x7f9fec0b4d60 2024-03-25T14:22:14.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.639+0000 7f9fe27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:14.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.642+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293318817 0 0) 0x7f9fec138460 con 0x7f9fec14be90 2024-03-25T14:22:14.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.642+0000 7f9fe27fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:14.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.642+0000 7f9fe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:14.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.642+0000 7f9fe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:14.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.642+0000 7f9fe27fc640 10 monclient(hunting): my global_id is 4322 2024-03-25T14:22:14.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.642+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd0004eb0 con 0x7f9fec14be90 2024-03-25T14:22:14.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.645+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255540897 0 0) 0x7f9fec143d40 con 0x7f9fec0b4d60 2024-03-25T14:22:14.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.645+0000 7f9fe27fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:14.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.645+0000 7f9fe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:14.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.645+0000 7f9fe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:14.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.645+0000 7f9fe27fc640 10 monclient(hunting): my global_id is 4326 2024-03-25T14:22:14.648 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.645+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec138460 con 0x7f9fec0b4d60 2024-03-25T14:22:14.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.659+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462191571 0 0) 0x7f9fec140f00 con 0x7f9fec0b1730 2024-03-25T14:22:14.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.659+0000 7f9fe27fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:14.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.659+0000 7f9fe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:14.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.659+0000 7f9fe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:14.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.659+0000 7f9fe27fc640 10 monclient(hunting): my global_id is 4339 2024-03-25T14:22:14.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.659+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec143d40 con 0x7f9fec0b1730 2024-03-25T14:22:14.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.666+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064331197 0 0) 0x7f9fec138460 con 0x7f9fec0b4d60 2024-03-25T14:22:14.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.666+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fec140f00 con 0x7f9fec0b4d60 2024-03-25T14:22:14.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.666+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9fdc001950 con 0x7f9fec0b4d60 2024-03-25T14:22:14.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.666+0000 7f9fe27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.666+0000 7f9fe27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.666+0000 7f9fe27fc640 10 monclient(hunting): dump: 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:14.670 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:14.671 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:14.671 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:14.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.666+0000 7f9fe27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.667+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 407165834 0 0) 0x7f9fd0004eb0 con 0x7f9fec14be90 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.667+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fec138460 con 0x7f9fec14be90 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.667+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9fe4001c70 con 0x7f9fec14be90 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.667+0000 7f9fe27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.667+0000 7f9fe27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.667+0000 7f9fe27fc640 10 monclient(hunting): dump: 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:14.672 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.673 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.673 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:14.673 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:14.673 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:14.673 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:14.673 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:14.673 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.673+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1474455531 0 0) 0x7f9fec143d40 con 0x7f9fec0b1730 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.673+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fd0004eb0 con 0x7f9fec0b1730 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.673+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9fe8001ae0 con 0x7f9fec0b1730 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.673+0000 7f9fe27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.673+0000 7f9fe27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.673+0000 7f9fe27fc640 10 monclient(hunting): dump: 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.676 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:14.677 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:14.677 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:14.677 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:14.677 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:14.677 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:14.677 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:14.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1273025953 0 0) 0x7f9fec138460 con 0x7f9fec14be90 2024-03-25T14:22:14.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 >> v1:172.21.15.153:6789/0 conn(0x7f9fec0b1730 legacy=0x7f9fec137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:14.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 >> v1:172.21.15.63:6789/0 conn(0x7f9fec0b4d60 legacy=0x7f9fec142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:14.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 10 monclient: _finish_hunting 0 2024-03-25T14:22:14.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 1 monclient: found mon.c 2024-03-25T14:22:14.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:14.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:14.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec141130 con 0x7f9fec14be90 2024-03-25T14:22:14.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 10 monclient: _finish_auth 0 2024-03-25T14:22:14.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 10 monclient: _check_auth_tickets 2024-03-25T14:22:14.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9fe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:14.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 5 monclient: authenticate success, global_id 4322 2024-03-25T14:22:14.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 10 monclient: _renew_subs 2024-03-25T14:22:14.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:14.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec137660 con 0x7f9fec14be90 2024-03-25T14:22:14.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:22:14.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 10 monclient: _renew_subs 2024-03-25T14:22:14.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:14.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec143030 con 0x7f9fec14be90 2024-03-25T14:22:14.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.686+0000 7f9ff3bf4640 1 librados: init done 2024-03-25T14:22:14.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.687+0000 7f9ff3bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:14.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.687+0000 7f9ff3bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:14.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.687+0000 7f9ff3bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:14.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:14.687+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec1d79d0 con 0x7f9fec14be90 2024-03-25T14:22:15.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.007+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe4003990 con 0x7f9fec14be90 2024-03-25T14:22:15.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.007+0000 7f9fe27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.007+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9fe4003d20 con 0x7f9fec14be90 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.007+0000 7f9fe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.007+0000 7f9fe27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.007+0000 7f9fe27fc640 10 monclient: dump: 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:15.011 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:15.012 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:15.012 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:15.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.007+0000 7f9fe3fff640 4 set_mon_vals no callback set 2024-03-25T14:22:15.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.008+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f9fe4003f40 con 0x7f9fec14be90 2024-03-25T14:22:15.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.009+0000 7f9ff1990640 10 monclient: get_auth_request con 0x7f9fd006e110 auth_method 0 2024-03-25T14:22:15.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.009+0000 7f9fe27fc640 10 client.4322.objecter ms_handle_connect 0x7f9fd006e110 2024-03-25T14:22:15.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f9fe4054a10 con 0x7f9fec14be90 2024-03-25T14:22:15.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 10 client.4322.objecter ms_dispatch 0x7f9fec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:22:15.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:22:15.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:22:15.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-25T14:22:15.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fe4050520 con 0x7f9fec14be90 2024-03-25T14:22:15.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:15.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.014+0000 7f9fe27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:22:15.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.264+0000 7f9ff3bf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:15.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.264+0000 7f9ff3bf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:15.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.264+0000 7f9ff3bf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:15.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.264+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fec0627b0 con 0x7f9fec14be90 2024-03-25T14:22:15.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.312+0000 7f9fe27fc640 1 -- 172.21.15.63:0/187378389 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f9fe4052450 con 0x7f9fec14be90 2024-03-25T14:22:15.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.312+0000 7f9fe27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:15.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.312+0000 7f9fe27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:22:15.316 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:22:15.316 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:22:15.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 10 librados: watch_flush enter 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 10 librados: watch_flush exit 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 10 client.4322.objecter successfully canceled tick 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 >> v1:172.21.15.153:6800/1688255552 conn(0x7f9fd006e110 legacy=0x7f9fd00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 10 monclient: shutdown 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 >> v1:172.21.15.63:6790/0 conn(0x7f9fec14be90 legacy=0x7f9fec1d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.314+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 shutdown_connections 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.315+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 >> 172.21.15.63:0/187378389 conn(0x7f9fec0617a0 msgr2=0x7f9fec0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.315+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 shutdown_connections 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.315+0000 7f9ff3bf4640 1 -- 172.21.15.63:0/187378389 wait complete. 2024-03-25T14:22:15.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:15.315+0000 7f9ff3bf4640 1 librados: shutdown 2024-03-25T14:22:15.330 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:22:22.137 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.249+0000 7fe0a822a640 10 monclient: get_monmap_and_config 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.249+0000 7fe0a822a640 10 monclient: build_initial_monmap 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.249+0000 7fe0a822a640 10 monclient: monmap: 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:22:22.250394+0000 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:22:22.250394+0000 2024-03-25T14:22:22.252 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:22.253 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:22.253 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:22.253 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:22:22.253 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:22:22.253 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.249+0000 7fe0a822a640 1 Processor -- start 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 1 -- start start 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 10 monclient: init 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a00a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a01430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:22.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0142520 con 0x7fe0a013f710 2024-03-25T14:22:22.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.250+0000 7fe0a822a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00b28a0 con 0x7fe0a01430c0 2024-03-25T14:22:22.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.251+0000 7fe0a822a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a00b2a30 con 0x7fe0a00a98f0 2024-03-25T14:22:22.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.251+0000 7fe0a822a640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:22.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.251+0000 7fe0a822a640 10 monclient(hunting): authenticate will time out at 1505.395575s 2024-03-25T14:22:22.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.260+0000 7fe0a57c5640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe0a013f710 0x7fe0a01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51422/0 (socket says 172.21.15.63:51422) 2024-03-25T14:22:22.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.260+0000 7fe0a57c5640 1 -- 172.21.15.63:0/3298393175 learned_addr learned my addr 172.21.15.63:0/3298393175 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:22.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.266+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535585784 0 0) 0x7fe0a00b28a0 con 0x7fe0a01430c0 2024-03-25T14:22:22.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.266+0000 7fe0a4fc4640 10 monclient(hunting): _negotiate 2024-03-25T14:22:22.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.266+0000 7fe0a4fc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:22.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.266+0000 7fe0a4fc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:22.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.266+0000 7fe0a4fc4640 10 monclient(hunting): my global_id is 4342 2024-03-25T14:22:22.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.266+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c003520 con 0x7fe0a01430c0 2024-03-25T14:22:22.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.274+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217159922 0 0) 0x7fe0a00b2a30 con 0x7fe0a00a98f0 2024-03-25T14:22:22.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.274+0000 7fe0a4fc4640 10 monclient(hunting): _negotiate 2024-03-25T14:22:22.277 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.274+0000 7fe0a4fc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:22.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.274+0000 7fe0a4fc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:22.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.274+0000 7fe0a4fc4640 10 monclient(hunting): my global_id is 4325 2024-03-25T14:22:22.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.274+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a00b28a0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.277+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533883652 0 0) 0x7fe0a0142520 con 0x7fe0a013f710 2024-03-25T14:22:22.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.277+0000 7fe0a4fc4640 10 monclient(hunting): _negotiate 2024-03-25T14:22:22.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.277+0000 7fe0a4fc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:22.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.277+0000 7fe0a4fc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:22.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.277+0000 7fe0a4fc4640 10 monclient(hunting): my global_id is 4329 2024-03-25T14:22:22.280 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.277+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a00b2a30 con 0x7fe0a013f710 2024-03-25T14:22:22.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402123670 0 0) 0x7fe08c003520 con 0x7fe0a01430c0 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 >> v1:172.21.15.63:6790/0 conn(0x7fe0a00a98f0 legacy=0x7fe0a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 >> v1:172.21.15.63:6789/0 conn(0x7fe0a013f710 legacy=0x7fe0a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 10 monclient: _finish_hunting 0 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 1 monclient: found mon.noname-c 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a00b44c0 con 0x7fe0a01430c0 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 10 monclient: _finish_auth 0 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 10 monclient: _check_auth_tickets 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe098001a10 con 0x7fe0a01430c0 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a822a640 5 monclient: authenticate success, global_id 4342 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a822a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:22.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a4fc4640 10 monclient: dump: 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:22.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.281+0000 7fe0a822a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe098001b70 con 0x7fe0a01430c0 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a4fc4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a4fc4640 1 -- 172.21.15.63:0/3298393175 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe098001f80 con 0x7fe0a01430c0 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a4fc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a4fc4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a4fc4640 10 monclient: dump: 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:22.303 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a822a640 10 monclient: get_monmap_and_config success 2024-03-25T14:22:22.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.300+0000 7fe0a822a640 4 set_mon_vals no callback set 2024-03-25T14:22:22.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.301+0000 7fe0a822a640 10 monclient: shutdown 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.301+0000 7fe0a822a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.301+0000 7fe0a822a640 1 -- 172.21.15.63:0/3298393175 >> v1:172.21.15.153:6789/0 conn(0x7fe0a01430c0 legacy=0x7fe0a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.301+0000 7fe0a822a640 1 -- 172.21.15.63:0/3298393175 shutdown_connections 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.301+0000 7fe0a822a640 1 -- 172.21.15.63:0/3298393175 >> 172.21.15.63:0/3298393175 conn(0x7fe0a00b7ea0 msgr2=0x7fe0a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.301+0000 7fe0a822a640 1 -- 172.21.15.63:0/3298393175 shutdown_connections 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.301+0000 7fe0a822a640 1 -- 172.21.15.63:0/3298393175 wait complete. 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 10 monclient: build_initial_monmap 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 10 monclient: monmap: 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:22:22.306 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 1 librados: starting msgr at 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 1 librados: starting objecter 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 1 Processor -- start 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 1 -- start start 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 1 librados: setting wanted keys 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 1 librados: calling monclient init 2024-03-25T14:22:22.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.302+0000 7fe0a822a640 10 monclient: init 2024-03-25T14:22:22.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:22.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:22:22.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a00a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:22.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a01430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0135230 con 0x7fe0a01430c0 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0135da0 con 0x7fe0a013f710 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0135f30 con 0x7fe0a00a98f0 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.303+0000 7fe0a822a640 10 monclient(hunting): authenticate will time out at 1505.448147s 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.306+0000 7fe0a5fc6640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe0a00a98f0 0x7fe0a01d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:49814/0 (socket says 172.21.15.63:49814) 2024-03-25T14:22:22.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.306+0000 7fe0a5fc6640 1 -- 172.21.15.63:0/977014113 learned_addr learned my addr 172.21.15.63:0/977014113 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:22.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.312+0000 7fe0967fc640 10 client.?.objecter ms_handle_connect 0x7fe0a013f710 2024-03-25T14:22:22.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.312+0000 7fe0967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:22.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.313+0000 7fe0967fc640 10 client.?.objecter ms_handle_connect 0x7fe0a00a98f0 2024-03-25T14:22:22.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.313+0000 7fe0967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:22.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.322+0000 7fe0967fc640 10 client.?.objecter ms_handle_connect 0x7fe0a01430c0 2024-03-25T14:22:22.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.322+0000 7fe0967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:22.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.328+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970408465 0 0) 0x7fe0a0135f30 con 0x7fe0a00a98f0 2024-03-25T14:22:22.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.328+0000 7fe0967fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:22.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.328+0000 7fe0967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:22.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.328+0000 7fe0967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:22.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.328+0000 7fe0967fc640 10 monclient(hunting): my global_id is 4328 2024-03-25T14:22:22.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.328+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe084004eb0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.336+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2808072375 0 0) 0x7fe084004eb0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.336+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a0135f30 con 0x7fe0a00a98f0 2024-03-25T14:22:22.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.336+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe09c001a30 con 0x7fe0a00a98f0 2024-03-25T14:22:22.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.336+0000 7fe0967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:22.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.336+0000 7fe0967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:22:22.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.336+0000 7fe0967fc640 10 monclient(hunting): dump: 2024-03-25T14:22:22.339 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:22.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.336+0000 7fe0967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:22:22.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.338+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376149491 0 0) 0x7fe0a0135da0 con 0x7fe0a013f710 2024-03-25T14:22:22.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.338+0000 7fe0967fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:22.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.338+0000 7fe0967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:22.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.338+0000 7fe0967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:22.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.338+0000 7fe0967fc640 10 monclient(hunting): my global_id is 4345 2024-03-25T14:22:22.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.338+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe084004eb0 con 0x7fe0a013f710 2024-03-25T14:22:22.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.343+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734638043 0 0) 0x7fe0a0135230 con 0x7fe0a01430c0 2024-03-25T14:22:22.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.343+0000 7fe0967fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:22.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.343+0000 7fe0967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:22.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.343+0000 7fe0967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:22.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.343+0000 7fe0967fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T14:22:22.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.343+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0135da0 con 0x7fe0a01430c0 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3196065423 0 0) 0x7fe0a0135f30 con 0x7fe0a00a98f0 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 >> v1:172.21.15.153:6789/0 conn(0x7fe0a013f710 legacy=0x7fe0a01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 >> v1:172.21.15.63:6789/0 conn(0x7fe0a01430c0 legacy=0x7fe0a01db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 10 monclient: _finish_hunting 0 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 1 monclient: found mon.c 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a01360c0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 10 monclient: _finish_auth 0 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 10 monclient: _check_auth_tickets 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0a822a640 5 monclient: authenticate success, global_id 4328 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0a822a640 10 monclient: _renew_subs 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0a822a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.351+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a01d5000 con 0x7fe0a00a98f0 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.352+0000 7fe0a822a640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.352+0000 7fe0a822a640 10 monclient: _renew_subs 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.352+0000 7fe0a822a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.352+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a01d5350 con 0x7fe0a00a98f0 2024-03-25T14:22:22.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.352+0000 7fe0a822a640 1 librados: init done 2024-03-25T14:22:22.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.353+0000 7fe0a822a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:22.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.353+0000 7fe0a822a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:22.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.353+0000 7fe0a822a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:22.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.353+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a0135090 con 0x7fe0a00a98f0 2024-03-25T14:22:22.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.661+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe09c001cd0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.661+0000 7fe0967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.661+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe09c0038c0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.661+0000 7fe0967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.661+0000 7fe0967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.661+0000 7fe0967fc640 10 monclient: dump: 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:22.665 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:22.666 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:22.666 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:22.666 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:22.666 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:22.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.661+0000 7fe097fff640 4 set_mon_vals no callback set 2024-03-25T14:22:22.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.662+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fe09c003ae0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.663+0000 7fe0a57c5640 10 monclient: get_auth_request con 0x7fe08406de80 auth_method 0 2024-03-25T14:22:22.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.663+0000 7fe0967fc640 10 client.4328.objecter ms_handle_connect 0x7fe08406de80 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fe09c054760 con 0x7fe0a00a98f0 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 10 client.4328.objecter ms_dispatch 0x7fe0a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe09c050270 con 0x7fe0a00a98f0 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:22.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.669+0000 7fe0967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:22:22.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.920+0000 7fe0a822a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:22.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.920+0000 7fe0a822a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:22.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.920+0000 7fe0a822a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:22.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.920+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0a013b700 con 0x7fe0a00a98f0 2024-03-25T14:22:22.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.948+0000 7fe0967fc640 1 -- 172.21.15.63:0/977014113 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fe09c0521a0 con 0x7fe0a00a98f0 2024-03-25T14:22:22.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.948+0000 7fe0967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:22.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.948+0000 7fe0967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:22:22.951 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:22:22.951 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:22:22.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.949+0000 7fe0a822a640 10 librados: watch_flush enter 2024-03-25T14:22:22.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 10 librados: watch_flush exit 2024-03-25T14:22:22.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 10 client.4328.objecter successfully canceled tick 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 >> v1:172.21.15.153:6800/1688255552 conn(0x7fe08406de80 legacy=0x7fe084070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 10 monclient: shutdown 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 >> v1:172.21.15.63:6790/0 conn(0x7fe0a00a98f0 legacy=0x7fe0a01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 shutdown_connections 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 >> 172.21.15.63:0/977014113 conn(0x7fe0a00b7ea0 msgr2=0x7fe0a014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 shutdown_connections 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 1 -- 172.21.15.63:0/977014113 wait complete. 2024-03-25T14:22:22.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:22.950+0000 7fe0a822a640 1 librados: shutdown 2024-03-25T14:22:22.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:22:26.210 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:04.723 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.219 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:04.732 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.252 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:04.766 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.260 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:04.720 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.272 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:04.732 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.281 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:04.795 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.286 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:04.799 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.302 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:04.763 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:26.321 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:04.834 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:29.772 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:22:29.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.881+0000 7fba6fd15640 10 monclient: get_monmap_and_config 2024-03-25T14:22:29.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.881+0000 7fba6fd15640 10 monclient: build_initial_monmap 2024-03-25T14:22:29.884 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.881+0000 7fba6fd15640 10 monclient: monmap: 2024-03-25T14:22:29.884 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:29.884 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:29.884 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:22:29.882318+0000 2024-03-25T14:22:29.884 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:22:29.882318+0000 2024-03-25T14:22:29.885 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:29.885 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:29.885 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:29.885 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:22:29.885 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:22:29.885 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:29.886 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.881+0000 7fba6fd15640 1 Processor -- start 2024-03-25T14:22:29.886 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 1 -- start start 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient: init 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient(hunting): picked mon.noname-b con 0x7fba680b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient(hunting): picked mon.noname-c con 0x7fba680b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient(hunting): picked mon.noname-a con 0x7fba6814be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680af940 con 0x7fba6814be90 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680b0a70 con 0x7fba680b1730 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680b0d50 con 0x7fba680b4d60 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:29.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.882+0000 7fba6fd15640 10 monclient(hunting): authenticate will time out at 1513.027375s 2024-03-25T14:22:29.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.886+0000 7fba6e2b2640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fba6814be90 0x7fba6814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:34604/0 (socket says 172.21.15.63:34604) 2024-03-25T14:22:29.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.886+0000 7fba6e2b2640 1 -- 172.21.15.63:0/1315322063 learned_addr learned my addr 172.21.15.63:0/1315322063 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:29.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.891+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803030600 0 0) 0x7fba680af940 con 0x7fba6814be90 2024-03-25T14:22:29.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.892+0000 7fba6caaf640 10 monclient(hunting): _negotiate 2024-03-25T14:22:29.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.892+0000 7fba6caaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:29.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.892+0000 7fba6caaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:29.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.892+0000 7fba6caaf640 10 monclient(hunting): my global_id is 4335 2024-03-25T14:22:29.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.892+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba54003520 con 0x7fba6814be90 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877745792 0 0) 0x7fba54003520 con 0x7fba6814be90 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 >> v1:172.21.15.63:6790/0 conn(0x7fba680b4d60 legacy=0x7fba6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 >> v1:172.21.15.153:6789/0 conn(0x7fba680b1730 legacy=0x7fba680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 10 monclient: _finish_hunting 0 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 1 monclient: found mon.noname-a 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:22:29.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680b1030 con 0x7fba6814be90 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 10 monclient: _finish_auth 0 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 10 monclient: _check_auth_tickets 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fba60001a10 con 0x7fba6814be90 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6fd15640 5 monclient: authenticate success, global_id 4335 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6fd15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6caaf640 10 monclient: dump: 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:29.905 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:29.906 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:29.906 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:29.906 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:29.906 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:29.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.901+0000 7fba6fd15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:29.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba60001be0 con 0x7fba6814be90 2024-03-25T14:22:29.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6caaf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:29.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6caaf640 1 -- 172.21.15.63:0/1315322063 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fba60002030 con 0x7fba6814be90 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6caaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6caaf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6caaf640 10 monclient: dump: 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:29.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6fd15640 10 monclient: get_monmap_and_config success 2024-03-25T14:22:29.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.912+0000 7fba6fd15640 4 set_mon_vals no callback set 2024-03-25T14:22:29.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.913+0000 7fba6fd15640 10 monclient: shutdown 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.913+0000 7fba6fd15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.913+0000 7fba6fd15640 1 -- 172.21.15.63:0/1315322063 >> v1:172.21.15.63:6789/0 conn(0x7fba6814be90 legacy=0x7fba6814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.913+0000 7fba6fd15640 1 -- 172.21.15.63:0/1315322063 shutdown_connections 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.913+0000 7fba6fd15640 1 -- 172.21.15.63:0/1315322063 >> 172.21.15.63:0/1315322063 conn(0x7fba680617a0 msgr2=0x7fba68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.913+0000 7fba6fd15640 1 -- 172.21.15.63:0/1315322063 shutdown_connections 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.913+0000 7fba6fd15640 1 -- 172.21.15.63:0/1315322063 wait complete. 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 10 monclient: build_initial_monmap 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 10 monclient: monmap: 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:22:29.919 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 1 librados: starting msgr at 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 1 librados: starting objecter 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 1 Processor -- start 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 1 -- start start 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 1 librados: setting wanted keys 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 1 librados: calling monclient init 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.914+0000 7fba6fd15640 10 monclient: init 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:22:29.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 10 monclient(hunting): picked mon.noname-b con 0x7fba680b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:29.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 10 monclient(hunting): picked mon.noname-a con 0x7fba680b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:29.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 10 monclient(hunting): picked mon.noname-c con 0x7fba6814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:29.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba681d79c0 con 0x7fba680b4d60 2024-03-25T14:22:29.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba681d7690 con 0x7fba680b1730 2024-03-25T14:22:29.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba681d8560 con 0x7fba6814be90 2024-03-25T14:22:29.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:29.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.915+0000 7fba6fd15640 10 monclient(hunting): authenticate will time out at 1513.060119s 2024-03-25T14:22:29.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.919+0000 7fba6dab1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fba680b1730 0x7fba681d7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:35656/0 (socket says 172.21.15.63:35656) 2024-03-25T14:22:29.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.919+0000 7fba6dab1640 1 -- 172.21.15.63:0/3489395259 learned_addr learned my addr 172.21.15.63:0/3489395259 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:29.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.924+0000 7fba5e7fc640 10 client.?.objecter ms_handle_connect 0x7fba680b1730 2024-03-25T14:22:29.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.924+0000 7fba5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:29.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.932+0000 7fba5e7fc640 10 client.?.objecter ms_handle_connect 0x7fba6814be90 2024-03-25T14:22:29.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.932+0000 7fba5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:29.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.933+0000 7fba5e7fc640 10 client.?.objecter ms_handle_connect 0x7fba680b4d60 2024-03-25T14:22:29.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.933+0000 7fba5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:29.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.936+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648078226 0 0) 0x7fba681d7690 con 0x7fba680b1730 2024-03-25T14:22:29.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.936+0000 7fba5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:29.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.936+0000 7fba5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:29.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.936+0000 7fba5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:29.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.936+0000 7fba5e7fc640 10 monclient(hunting): my global_id is 4351 2024-03-25T14:22:29.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.936+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c004eb0 con 0x7fba680b1730 2024-03-25T14:22:29.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.941+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492085826 0 0) 0x7fba681d8560 con 0x7fba6814be90 2024-03-25T14:22:29.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.941+0000 7fba5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:29.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.941+0000 7fba5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:29.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.941+0000 7fba5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:29.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.941+0000 7fba5e7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T14:22:29.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.941+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba681d7690 con 0x7fba6814be90 2024-03-25T14:22:29.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.953+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2355186421 0 0) 0x7fba4c004eb0 con 0x7fba680b1730 2024-03-25T14:22:29.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.953+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba681d8560 con 0x7fba680b1730 2024-03-25T14:22:29.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.953+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fba64004080 con 0x7fba680b1730 2024-03-25T14:22:29.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.953+0000 7fba5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:29.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.953+0000 7fba5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:22:29.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.953+0000 7fba5e7fc640 10 monclient(hunting): dump: 2024-03-25T14:22:29.956 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:29.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.953+0000 7fba5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:22:29.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.956+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1442005934 0 0) 0x7fba681d7690 con 0x7fba6814be90 2024-03-25T14:22:29.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.956+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba4c004eb0 con 0x7fba6814be90 2024-03-25T14:22:29.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.956+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fba60001ef0 con 0x7fba6814be90 2024-03-25T14:22:29.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.956+0000 7fba5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:29.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.956+0000 7fba5e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:29.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.956+0000 7fba5e7fc640 10 monclient(hunting): dump: 2024-03-25T14:22:29.959 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:29.960 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:29.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.957+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568024128 0 0) 0x7fba681d79c0 con 0x7fba680b4d60 2024-03-25T14:22:29.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.957+0000 7fba5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:22:29.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.957+0000 7fba5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:29.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.957+0000 7fba5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:29.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.957+0000 7fba5e7fc640 10 monclient(hunting): my global_id is 4338 2024-03-25T14:22:29.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.957+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba681d7690 con 0x7fba680b4d60 2024-03-25T14:22:29.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2937142932 0 0) 0x7fba681d8560 con 0x7fba680b1730 2024-03-25T14:22:29.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 >> v1:172.21.15.63:6790/0 conn(0x7fba6814be90 legacy=0x7fba681e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:29.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 >> v1:172.21.15.63:6789/0 conn(0x7fba680b4d60 legacy=0x7fba68137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 1 monclient: found mon.b 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba68142e30 con 0x7fba680b1730 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 5 monclient: authenticate success, global_id 4351 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 10 monclient: _renew_subs 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba68143080 con 0x7fba680b1730 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:22:29.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 10 monclient: _renew_subs 2024-03-25T14:22:29.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:29.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba68142b50 con 0x7fba680b1730 2024-03-25T14:22:29.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.969+0000 7fba6fd15640 1 librados: init done 2024-03-25T14:22:29.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.970+0000 7fba6fd15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:29.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.970+0000 7fba6fd15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:29.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.970+0000 7fba6fd15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:29.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:29.970+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba680627b0 con 0x7fba680b1730 2024-03-25T14:22:30.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.284+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba64000ed0 con 0x7fba680b1730 2024-03-25T14:22:30.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.284+0000 7fba5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.284+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fba640012a0 con 0x7fba680b1730 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.284+0000 7fba5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.284+0000 7fba5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.284+0000 7fba5e7fc640 10 monclient: dump: 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:30.288 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:30.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.284+0000 7fba5ffff640 4 set_mon_vals no callback set 2024-03-25T14:22:30.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.285+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fba640014c0 con 0x7fba680b1730 2024-03-25T14:22:30.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.286+0000 7fba6d2b0640 10 monclient: get_auth_request con 0x7fba4c06df00 auth_method 0 2024-03-25T14:22:30.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.286+0000 7fba5e7fc640 10 client.4351.objecter ms_handle_connect 0x7fba4c06df00 2024-03-25T14:22:30.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fba64054750 con 0x7fba680b1730 2024-03-25T14:22:30.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 10 client.4351.objecter ms_dispatch 0x7fba68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:22:30.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:22:30.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:22:30.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T14:22:30.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba64050260 con 0x7fba680b1730 2024-03-25T14:22:30.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:30.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.290+0000 7fba5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:22:30.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.547+0000 7fba6fd15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:30.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.547+0000 7fba6fd15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:30.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.547+0000 7fba6fd15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:30.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.547+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba68069f40 con 0x7fba680b1730 2024-03-25T14:22:30.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.569+0000 7fba5e7fc640 1 -- 172.21.15.63:0/3489395259 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fba64052190 con 0x7fba680b1730 2024-03-25T14:22:30.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.569+0000 7fba5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:30.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.569+0000 7fba5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:22:30.573 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:22:30.573 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 10 librados: watch_flush enter 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 10 librados: watch_flush exit 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 10 client.4351.objecter successfully canceled tick 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 >> v1:172.21.15.153:6800/1688255552 conn(0x7fba4c06df00 legacy=0x7fba4c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 10 monclient: shutdown 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 >> v1:172.21.15.153:6789/0 conn(0x7fba680b1730 legacy=0x7fba681d7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 shutdown_connections 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 >> 172.21.15.63:0/3489395259 conn(0x7fba680617a0 msgr2=0x7fba6813c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 shutdown_connections 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 1 -- 172.21.15.63:0/3489395259 wait complete. 2024-03-25T14:22:30.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:30.572+0000 7fba6fd15640 1 librados: shutdown 2024-03-25T14:22:30.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:22:37.395 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.505+0000 7f423598d640 10 monclient: get_monmap_and_config 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.505+0000 7f423598d640 10 monclient: build_initial_monmap 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.505+0000 7f423598d640 10 monclient: monmap: 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:22:37.506970+0000 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:22:37.506970+0000 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:37.509 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:37.510 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:22:37.510 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:22:37.510 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.506+0000 7f423598d640 1 Processor -- start 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.506+0000 7f423598d640 1 -- start start 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.506+0000 7f423598d640 10 monclient: init 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 10 monclient(hunting): picked mon.noname-c con 0x7f42300a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 10 monclient(hunting): picked mon.noname-b con 0x7f423013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 10 monclient(hunting): picked mon.noname-a con 0x7f42301430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4230142520 con 0x7f42301430c0 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300b28a0 con 0x7f42300a98f0 2024-03-25T14:22:37.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300b2a30 con 0x7f423013f710 2024-03-25T14:22:37.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:37.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.507+0000 7f423598d640 10 monclient(hunting): authenticate will time out at 1520.652177s 2024-03-25T14:22:37.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.514+0000 7f422effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f42300a98f0 0x7f42300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:40472/0 (socket says 172.21.15.63:40472) 2024-03-25T14:22:37.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.514+0000 7f422effd640 1 -- 172.21.15.63:0/3798910173 learned_addr learned my addr 172.21.15.63:0/3798910173 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:37.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.529+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069659098 0 0) 0x7f4230142520 con 0x7f42301430c0 2024-03-25T14:22:37.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.529+0000 7f422dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:22:37.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.529+0000 7f422dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:37.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.529+0000 7f422dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:37.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.529+0000 7f422dffb640 10 monclient(hunting): my global_id is 4341 2024-03-25T14:22:37.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.529+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4218003520 con 0x7f42301430c0 2024-03-25T14:22:37.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.530+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121655371 0 0) 0x7f42300b28a0 con 0x7f42300a98f0 2024-03-25T14:22:37.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.530+0000 7f422dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:22:37.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.530+0000 7f422dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:37.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.530+0000 7f422dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:37.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.530+0000 7f422dffb640 10 monclient(hunting): my global_id is 4354 2024-03-25T14:22:37.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.530+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4230142520 con 0x7f42300a98f0 2024-03-25T14:22:37.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.539+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392661444 0 0) 0x7f42300b2a30 con 0x7f423013f710 2024-03-25T14:22:37.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.539+0000 7f422dffb640 10 monclient(hunting): _negotiate 2024-03-25T14:22:37.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.539+0000 7f422dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:37.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.539+0000 7f422dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:37.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.539+0000 7f422dffb640 10 monclient(hunting): my global_id is 4334 2024-03-25T14:22:37.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.539+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42300b28a0 con 0x7f423013f710 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 501063811 0 0) 0x7f4230142520 con 0x7f42300a98f0 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 >> v1:172.21.15.63:6790/0 conn(0x7f423013f710 legacy=0x7f42301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 >> v1:172.21.15.63:6789/0 conn(0x7f42301430c0 legacy=0x7f423014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 10 monclient: _finish_hunting 0 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 1 monclient: found mon.noname-c 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300b44c0 con 0x7f42300a98f0 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 10 monclient: _finish_auth 0 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 10 monclient: _check_auth_tickets 2024-03-25T14:22:37.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4220001670 con 0x7f42300a98f0 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f423598d640 5 monclient: authenticate success, global_id 4354 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f423598d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f422dffb640 10 monclient: dump: 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:37.548 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:37.549 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:37.549 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:37.549 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:37.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.543+0000 7f423598d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42200017d0 con 0x7f42300a98f0 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f422dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f422dffb640 1 -- 172.21.15.63:0/3798910173 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4220001be0 con 0x7f42300a98f0 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f423598d640 10 monclient: get_monmap_and_config success 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f423598d640 4 set_mon_vals no callback set 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f422dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f422dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f422dffb640 10 monclient: dump: 2024-03-25T14:22:37.556 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:37.557 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:37.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f423598d640 10 monclient: shutdown 2024-03-25T14:22:37.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f423598d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:37.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.553+0000 7f423598d640 1 -- 172.21.15.63:0/3798910173 >> v1:172.21.15.153:6789/0 conn(0x7f42300a98f0 legacy=0x7f42300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:37.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.554+0000 7f423598d640 1 -- 172.21.15.63:0/3798910173 shutdown_connections 2024-03-25T14:22:37.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.554+0000 7f423598d640 1 -- 172.21.15.63:0/3798910173 >> 172.21.15.63:0/3798910173 conn(0x7f42300b7ea0 msgr2=0x7f42300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.554+0000 7f423598d640 1 -- 172.21.15.63:0/3798910173 shutdown_connections 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.554+0000 7f423598d640 1 -- 172.21.15.63:0/3798910173 wait complete. 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.554+0000 7f423598d640 10 monclient: build_initial_monmap 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.554+0000 7f423598d640 10 monclient: monmap: 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:22:37.560 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 librados: starting msgr at 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 librados: starting objecter 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 Processor -- start 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 -- start start 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 librados: setting wanted keys 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 librados: calling monclient init 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient: init 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient(hunting): picked mon.noname-a con 0x7f42300a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient(hunting): picked mon.noname-b con 0x7f423013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient(hunting): picked mon.noname-c con 0x7f42301430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42301390f0 con 0x7f42300a98f0 2024-03-25T14:22:37.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4230138dd0 con 0x7f423013f710 2024-03-25T14:22:37.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42301346b0 con 0x7f42301430c0 2024-03-25T14:22:37.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:37.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.555+0000 7f423598d640 10 monclient(hunting): authenticate will time out at 1520.700475s 2024-03-25T14:22:37.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.570+0000 7f422e7fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f423013f710 0x7f42301386c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:40518/0 (socket says 172.21.15.63:40518) 2024-03-25T14:22:37.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.570+0000 7f422e7fc640 1 -- 172.21.15.63:0/4135139946 learned_addr learned my addr 172.21.15.63:0/4135139946 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:37.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.572+0000 7f4217fff640 10 client.?.objecter ms_handle_connect 0x7f423013f710 2024-03-25T14:22:37.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.572+0000 7f4217fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:37.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.581+0000 7f4217fff640 10 client.?.objecter ms_handle_connect 0x7f42300a98f0 2024-03-25T14:22:37.584 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.581+0000 7f4217fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:37.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.584+0000 7f4217fff640 10 client.?.objecter ms_handle_connect 0x7f42301430c0 2024-03-25T14:22:37.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.584+0000 7f4217fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:37.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.590+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316261476 0 0) 0x7f4230138dd0 con 0x7f423013f710 2024-03-25T14:22:37.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.590+0000 7f4217fff640 10 monclient(hunting): _negotiate 2024-03-25T14:22:37.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.590+0000 7f4217fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:37.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.590+0000 7f4217fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:37.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.590+0000 7f4217fff640 10 monclient(hunting): my global_id is 4357 2024-03-25T14:22:37.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.590+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c004eb0 con 0x7f423013f710 2024-03-25T14:22:37.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.594+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996776859 0 0) 0x7f42301390f0 con 0x7f42300a98f0 2024-03-25T14:22:37.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.594+0000 7f4217fff640 10 monclient(hunting): _negotiate 2024-03-25T14:22:37.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.594+0000 7f4217fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:37.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.594+0000 7f4217fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:37.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.594+0000 7f4217fff640 10 monclient(hunting): my global_id is 4344 2024-03-25T14:22:37.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.594+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4230138dd0 con 0x7f42300a98f0 2024-03-25T14:22:37.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.596+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560012852 0 0) 0x7f42301346b0 con 0x7f42301430c0 2024-03-25T14:22:37.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.596+0000 7f4217fff640 10 monclient(hunting): _negotiate 2024-03-25T14:22:37.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.596+0000 7f4217fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:37.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.596+0000 7f4217fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:37.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.596+0000 7f4217fff640 10 monclient(hunting): my global_id is 4337 2024-03-25T14:22:37.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.596+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42301390f0 con 0x7f42301430c0 2024-03-25T14:22:37.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.612+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265927188 0 0) 0x7f4230138dd0 con 0x7f42300a98f0 2024-03-25T14:22:37.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.612+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42301346b0 con 0x7f42300a98f0 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.612+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4220001ae0 con 0x7f42300a98f0 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.612+0000 7f4217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.612+0000 7f4217fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.612+0000 7f4217fff640 10 monclient(hunting): dump: 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:37.619 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.612+0000 7f4217fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.613+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1073856196 0 0) 0x7f421c004eb0 con 0x7f423013f710 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.613+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4230138dd0 con 0x7f423013f710 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.613+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4224001a10 con 0x7f423013f710 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.613+0000 7f4217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.613+0000 7f4217fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.613+0000 7f4217fff640 10 monclient(hunting): dump: 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:37.620 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.621 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.621 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:37.621 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:37.621 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:37.621 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:37.621 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:37.621 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.620+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334464307 0 0) 0x7f42301390f0 con 0x7f42301430c0 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.620+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f421c004eb0 con 0x7f42301430c0 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.620+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4228001a30 con 0x7f42301430c0 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.620+0000 7f4217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.620+0000 7f4217fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.620+0000 7f4217fff640 10 monclient(hunting): dump: 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:37.623 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.624 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:37.624 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:37.624 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:37.624 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:37.624 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:37.624 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:37.624 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1139888633 0 0) 0x7f42301346b0 con 0x7f42300a98f0 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 >> v1:172.21.15.63:6790/0 conn(0x7f42301430c0 legacy=0x7f4230133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 >> v1:172.21.15.153:6789/0 conn(0x7f423013f710 legacy=0x7f42301386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 10 monclient: _finish_hunting 0 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 1 monclient: found mon.a 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4230134840 con 0x7f42300a98f0 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 10 monclient: _finish_auth 0 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 10 monclient: _check_auth_tickets 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f4217fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:37.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 5 monclient: authenticate success, global_id 4344 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 10 monclient: _renew_subs 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4230134a90 con 0x7f42300a98f0 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 10 monclient: _renew_subs 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42301356f0 con 0x7f42300a98f0 2024-03-25T14:22:37.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.625+0000 7f423598d640 1 librados: init done 2024-03-25T14:22:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.626+0000 7f423598d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.626+0000 7f423598d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.626+0000 7f423598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:22:37.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:37.626+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42300627b0 con 0x7f42300a98f0 2024-03-25T14:22:38.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.002+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4220001210 con 0x7f42300a98f0 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.002+0000 7f4217fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.002+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4220004c30 con 0x7f42300a98f0 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.002+0000 7f4217fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.002+0000 7f4217fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.002+0000 7f4217fff640 10 monclient: dump: 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:38.006 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:38.007 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:38.007 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:38.007 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:38.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.002+0000 7f422d7fa640 4 set_mon_vals no callback set 2024-03-25T14:22:38.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.003+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f422001ac00 con 0x7f42300a98f0 2024-03-25T14:22:38.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.004+0000 7f422e7fc640 10 monclient: get_auth_request con 0x7f421c06e0f0 auth_method 0 2024-03-25T14:22:38.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.004+0000 7f4217fff640 10 client.4344.objecter ms_handle_connect 0x7f421c06e0f0 2024-03-25T14:22:38.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f4220054850 con 0x7f42300a98f0 2024-03-25T14:22:38.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 10 client.4344.objecter ms_dispatch 0x7f4230000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:22:38.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:22:38.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:22:38.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-25T14:22:38.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4220050360 con 0x7f42300a98f0 2024-03-25T14:22:38.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:38.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.011+0000 7f4217fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:22:38.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.267+0000 7f423598d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:38.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.267+0000 7f423598d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:38.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.267+0000 7f423598d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:22:38.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.267+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42301361b0 con 0x7f42300a98f0 2024-03-25T14:22:38.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.358+0000 7f4217fff640 1 -- 172.21.15.63:0/4135139946 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f4220052290 con 0x7f42300a98f0 2024-03-25T14:22:38.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.358+0000 7f4217fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:38.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.358+0000 7f4217fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:22:38.361 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:22:38.362 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 10 librados: watch_flush enter 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 10 librados: watch_flush exit 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 10 client.4344.objecter successfully canceled tick 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 >> v1:172.21.15.153:6800/1688255552 conn(0x7f421c06e0f0 legacy=0x7f421c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 10 monclient: shutdown 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 >> v1:172.21.15.63:6789/0 conn(0x7f42300a98f0 legacy=0x7f4230137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 shutdown_connections 2024-03-25T14:22:38.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 >> 172.21.15.63:0/4135139946 conn(0x7f42300b7ea0 msgr2=0x7f4230074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:38.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 shutdown_connections 2024-03-25T14:22:38.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 1 -- 172.21.15.63:0/4135139946 wait complete. 2024-03-25T14:22:38.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:38.360+0000 7f423598d640 1 librados: shutdown 2024-03-25T14:22:38.376 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:22:45.185 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.281+0000 7f0304e6b640 10 monclient: get_monmap_and_config 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.281+0000 7f0304e6b640 10 monclient: build_initial_monmap 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.281+0000 7f0304e6b640 10 monclient: monmap: 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:22:45.282519+0000 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:22:45.282519+0000 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:45.285 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:45.286 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:22:45.286 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:22:45.286 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:45.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.281+0000 7f0304e6b640 1 Processor -- start 2024-03-25T14:22:45.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 1 -- start start 2024-03-25T14:22:45.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient: init 2024-03-25T14:22:45.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:45.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient(hunting): picked mon.noname-a con 0x7f03000b10d0 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient(hunting): picked mon.noname-b con 0x7f03000b4360 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient(hunting): picked mon.noname-c con 0x7f03000a9470 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000a9ab0 con 0x7f03000b10d0 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000b2400 con 0x7f03000a9470 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000aa2a0 con 0x7f03000b4360 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:45.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.282+0000 7f0304e6b640 10 monclient(hunting): authenticate will time out at 1528.427374s 2024-03-25T14:22:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.294+0000 7f02fe59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f03000b10d0 0x7f03000b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:58734/0 (socket says 172.21.15.63:58734) 2024-03-25T14:22:45.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.294+0000 7f02fe59c640 1 -- 172.21.15.63:0/1163985931 learned_addr learned my addr 172.21.15.63:0/1163985931 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:45.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.311+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435793481 0 0) 0x7f03000a9ab0 con 0x7f03000b10d0 2024-03-25T14:22:45.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.311+0000 7f02fd59a640 10 monclient(hunting): _negotiate 2024-03-25T14:22:45.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.311+0000 7f02fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:45.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.311+0000 7f02fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:45.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.311+0000 7f02fd59a640 10 monclient(hunting): my global_id is 4347 2024-03-25T14:22:45.315 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.311+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e8003520 con 0x7f03000b10d0 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494144895 0 0) 0x7f02e8003520 con 0x7f03000b10d0 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 >> v1:172.21.15.63:6790/0 conn(0x7f03000b4360 legacy=0x7f03000b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 >> v1:172.21.15.153:6789/0 conn(0x7f03000a9470 legacy=0x7f030014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 10 monclient: _finish_hunting 0 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 1 monclient: found mon.noname-a 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000b14b0 con 0x7f03000b10d0 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 10 monclient: _finish_auth 0 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 10 monclient: _check_auth_tickets 2024-03-25T14:22:45.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02f0001690 con 0x7f03000b10d0 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f02fd59a640 10 monclient: dump: 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:45.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f0304e6b640 5 monclient: authenticate success, global_id 4347 2024-03-25T14:22:45.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.320+0000 7f0304e6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02f0001860 con 0x7f03000b10d0 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f02fd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f02fd59a640 1 -- 172.21.15.63:0/1163985931 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02f0001cb0 con 0x7f03000b10d0 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f0304e6b640 10 monclient: get_monmap_and_config success 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f0304e6b640 4 set_mon_vals no callback set 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f02fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f02fd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.333+0000 7f02fd59a640 10 monclient: dump: 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:45.336 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:45.337 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:45.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.334+0000 7f0304e6b640 10 monclient: shutdown 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.334+0000 7f0304e6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.334+0000 7f0304e6b640 1 -- 172.21.15.63:0/1163985931 >> v1:172.21.15.63:6789/0 conn(0x7f03000b10d0 legacy=0x7f03000b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.334+0000 7f0304e6b640 1 -- 172.21.15.63:0/1163985931 shutdown_connections 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.334+0000 7f0304e6b640 1 -- 172.21.15.63:0/1163985931 >> 172.21.15.63:0/1163985931 conn(0x7f03000617a0 msgr2=0x7f0300061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.334+0000 7f0304e6b640 1 -- 172.21.15.63:0/1163985931 shutdown_connections 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.334+0000 7f0304e6b640 1 -- 172.21.15.63:0/1163985931 wait complete. 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 10 monclient: build_initial_monmap 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 10 monclient: monmap: 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:22:45.339 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 1 librados: starting msgr at 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 1 librados: starting objecter 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 1 Processor -- start 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 1 -- start start 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 1 librados: setting wanted keys 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 1 librados: calling monclient init 2024-03-25T14:22:45.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.335+0000 7f0304e6b640 10 monclient: init 2024-03-25T14:22:45.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 10 monclient(hunting): picked mon.noname-b con 0x7f03000a9470 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 10 monclient(hunting): picked mon.noname-a con 0x7f03000b10d0 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 10 monclient(hunting): picked mon.noname-c con 0x7f03000b4360 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03001da9b0 con 0x7f03000b10d0 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030013ade0 con 0x7f03000a9470 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030013b050 con 0x7f03000b4360 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:45.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.336+0000 7f0304e6b640 10 monclient(hunting): authenticate will time out at 1528.481398s 2024-03-25T14:22:45.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.343+0000 7f02fe59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f03000a9470 0x7f03001dae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:44096/0 (socket says 172.21.15.63:44096) 2024-03-25T14:22:45.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.343+0000 7f02fe59c640 1 -- 172.21.15.63:0/1290361783 learned_addr learned my addr 172.21.15.63:0/1290361783 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:45.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.346+0000 7f02eeffd640 10 client.?.objecter ms_handle_connect 0x7f03000a9470 2024-03-25T14:22:45.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.346+0000 7f02eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:45.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.346+0000 7f02eeffd640 10 client.?.objecter ms_handle_connect 0x7f03000b4360 2024-03-25T14:22:45.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.346+0000 7f02eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:45.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.348+0000 7f02eeffd640 10 client.?.objecter ms_handle_connect 0x7f03000b10d0 2024-03-25T14:22:45.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.348+0000 7f02eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:45.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.357+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118920507 0 0) 0x7f030013b050 con 0x7f03000b4360 2024-03-25T14:22:45.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.357+0000 7f02eeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:22:45.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.357+0000 7f02eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:45.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.357+0000 7f02eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:45.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.357+0000 7f02eeffd640 10 monclient(hunting): my global_id is 4340 2024-03-25T14:22:45.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.357+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e0004eb0 con 0x7f03000b4360 2024-03-25T14:22:45.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.367+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283804030 0 0) 0x7f030013ade0 con 0x7f03000a9470 2024-03-25T14:22:45.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.367+0000 7f02eeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:22:45.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.367+0000 7f02eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:45.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.367+0000 7f02eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:45.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.367+0000 7f02eeffd640 10 monclient(hunting): my global_id is 4360 2024-03-25T14:22:45.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.367+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030013b050 con 0x7f03000a9470 2024-03-25T14:22:45.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.373+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094219160 0 0) 0x7f03001da9b0 con 0x7f03000b10d0 2024-03-25T14:22:45.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.373+0000 7f02eeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:22:45.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.373+0000 7f02eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:45.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.373+0000 7f02eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:45.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.373+0000 7f02eeffd640 10 monclient(hunting): my global_id is 4350 2024-03-25T14:22:45.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.373+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030013ade0 con 0x7f03000b10d0 2024-03-25T14:22:45.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.377+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2561484893 0 0) 0x7f030013b050 con 0x7f03000a9470 2024-03-25T14:22:45.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.377+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03001da9b0 con 0x7f03000a9470 2024-03-25T14:22:45.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.377+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02f0001b70 con 0x7f03000a9470 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.377+0000 7f02eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.377+0000 7f02eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.377+0000 7f02eeffd640 10 monclient(hunting): dump: 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:45.381 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:45.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.377+0000 7f02eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:22:45.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.386+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525472897 0 0) 0x7f02e0004eb0 con 0x7f03000b4360 2024-03-25T14:22:45.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.386+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f030013b050 con 0x7f03000b4360 2024-03-25T14:22:45.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.386+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02f8001b70 con 0x7f03000b4360 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.386+0000 7f02eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.386+0000 7f02eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.386+0000 7f02eeffd640 10 monclient(hunting): dump: 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.390 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:45.391 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:45.391 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:45.391 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:45.391 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:45.391 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:45.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.390+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2082871121 0 0) 0x7f030013b050 con 0x7f03000b4360 2024-03-25T14:22:45.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.390+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 >> v1:172.21.15.153:6789/0 conn(0x7f03000a9470 legacy=0x7f03001dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:45.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 >> v1:172.21.15.63:6789/0 conn(0x7f03000b10d0 legacy=0x7f0300141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 10 monclient: _finish_hunting 0 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 1 monclient: found mon.c 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03001db550 con 0x7f03000b4360 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 10 monclient: _finish_auth 0 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 10 monclient: _check_auth_tickets 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f02eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 5 monclient: authenticate success, global_id 4340 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 10 monclient: _renew_subs 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03001e2bc0 con 0x7f03000b4360 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:22:45.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 10 monclient: _renew_subs 2024-03-25T14:22:45.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:45.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03001e2f30 con 0x7f03000b4360 2024-03-25T14:22:45.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.391+0000 7f0304e6b640 1 librados: init done 2024-03-25T14:22:45.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.392+0000 7f0304e6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:45.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.392+0000 7f0304e6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:45.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.392+0000 7f0304e6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:45.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.392+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03000ab3e0 con 0x7f03000b4360 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.702+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02f8002080 con 0x7f03000b4360 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.702+0000 7f02eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.702+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02f8003b10 con 0x7f03000b4360 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.702+0000 7f02eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.702+0000 7f02eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.702+0000 7f02eeffd640 10 monclient: dump: 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:45.706 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:45.707 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:45.707 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:45.707 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:45.707 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:45.707 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:45.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.702+0000 7f02fcd99640 4 set_mon_vals no callback set 2024-03-25T14:22:45.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.703+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f02f8019ae0 con 0x7f03000b4360 2024-03-25T14:22:45.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.704+0000 7f02fe59c640 10 monclient: get_auth_request con 0x7f02e006dfb0 auth_method 0 2024-03-25T14:22:45.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.704+0000 7f02eeffd640 10 client.4340.objecter ms_handle_connect 0x7f02e006dfb0 2024-03-25T14:22:45.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f02f8053890 con 0x7f03000b4360 2024-03-25T14:22:45.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 10 client.4340.objecter ms_dispatch 0x7f0300000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:22:45.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:22:45.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:22:45.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T14:22:45.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02f804f1d0 con 0x7f03000b4360 2024-03-25T14:22:45.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:45.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.710+0000 7f02eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:22:45.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.956+0000 7f0304e6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:45.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.956+0000 7f0304e6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:45.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.956+0000 7f0304e6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:22:45.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.956+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03000627b0 con 0x7f03000b4360 2024-03-25T14:22:46.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.997+0000 7f02eeffd640 1 -- 172.21.15.63:0/1290361783 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f02f80512d0 con 0x7f03000b4360 2024-03-25T14:22:46.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.997+0000 7f02eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:46.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:45.997+0000 7f02eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:22:46.001 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:22:46.002 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:22:46.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 10 librados: watch_flush enter 2024-03-25T14:22:46.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 10 librados: watch_flush exit 2024-03-25T14:22:46.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T14:22:46.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 10 client.4340.objecter successfully canceled tick 2024-03-25T14:22:46.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 >> v1:172.21.15.153:6800/1688255552 conn(0x7f02e006dfb0 legacy=0x7f02e0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:46.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 10 monclient: shutdown 2024-03-25T14:22:46.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:46.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 >> v1:172.21.15.63:6790/0 conn(0x7f03000b4360 legacy=0x7f0300142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:46.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 shutdown_connections 2024-03-25T14:22:46.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 >> 172.21.15.63:0/1290361783 conn(0x7f03000617a0 msgr2=0x7f0300145e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:46.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 shutdown_connections 2024-03-25T14:22:46.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.000+0000 7f0304e6b640 1 -- 172.21.15.63:0/1290361783 wait complete. 2024-03-25T14:22:46.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:46.001+0000 7f0304e6b640 1 librados: shutdown 2024-03-25T14:22:46.015 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:22:46.281 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:24.795 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.292 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:24.806 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.325 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:24.838 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.329 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:24.790 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.352 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:24.866 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.353 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:24.814 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.366 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:24.880 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.379 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:24.840 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:46.393 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:24.907 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:22:52.822 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:22:52.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.931+0000 7fd5836cf640 10 monclient: get_monmap_and_config 2024-03-25T14:22:52.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.931+0000 7fd5836cf640 10 monclient: build_initial_monmap 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.931+0000 7fd5836cf640 10 monclient: monmap: 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:22:52.932219+0000 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:22:52.932219+0000 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:22:52.935 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:52.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.931+0000 7fd5836cf640 1 Processor -- start 2024-03-25T14:22:52.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.932+0000 7fd5836cf640 1 -- start start 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.932+0000 7fd5836cf640 10 monclient: init 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.932+0000 7fd5836cf640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.932+0000 7fd5836cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.932+0000 7fd5836cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd57c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.933+0000 7fd5836cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd57c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.933+0000 7fd5836cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd57c1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.933+0000 7fd5836cf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c142520 con 0x7fd57c13f710 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.933+0000 7fd5836cf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0b28a0 con 0x7fd57c1430c0 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.933+0000 7fd5836cf640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0b2a30 con 0x7fd57c0a98f0 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.933+0000 7fd5836cf640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:52.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.933+0000 7fd5836cf640 10 monclient(hunting): authenticate will time out at 1536.077594s 2024-03-25T14:22:52.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.937+0000 7fd580c6a640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd57c13f710 0x7fd57c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:58746/0 (socket says 172.21.15.63:58746) 2024-03-25T14:22:52.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.937+0000 7fd580c6a640 1 -- 172.21.15.63:0/328580401 learned_addr learned my addr 172.21.15.63:0/328580401 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:52.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.948+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443548729 0 0) 0x7fd57c142520 con 0x7fd57c13f710 2024-03-25T14:22:52.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.948+0000 7fd573fff640 10 monclient(hunting): _negotiate 2024-03-25T14:22:52.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.948+0000 7fd573fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:52.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.948+0000 7fd573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:52.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.948+0000 7fd573fff640 10 monclient(hunting): my global_id is 4353 2024-03-25T14:22:52.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.948+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd568003520 con 0x7fd57c13f710 2024-03-25T14:22:52.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 990418890 0 0) 0x7fd568003520 con 0x7fd57c13f710 2024-03-25T14:22:52.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 >> v1:172.21.15.63:6790/0 conn(0x7fd57c0a98f0 legacy=0x7fd57c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:52.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 >> v1:172.21.15.153:6789/0 conn(0x7fd57c1430c0 legacy=0x7fd57c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:52.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 10 monclient: _finish_hunting 0 2024-03-25T14:22:52.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 1 monclient: found mon.noname-a 2024-03-25T14:22:52.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0b44c0 con 0x7fd57c13f710 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 10 monclient: _finish_auth 0 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 10 monclient: _check_auth_tickets 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd56c001690 con 0x7fd57c13f710 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd573fff640 10 monclient: dump: 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:52.959 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd5836cf640 5 monclient: authenticate success, global_id 4353 2024-03-25T14:22:52.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.955+0000 7fd5836cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:22:52.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd56c001870 con 0x7fd57c13f710 2024-03-25T14:22:52.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd573fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:52.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd573fff640 1 -- 172.21.15.63:0/328580401 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd56c001cc0 con 0x7fd57c13f710 2024-03-25T14:22:52.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd5836cf640 10 monclient: get_monmap_and_config success 2024-03-25T14:22:52.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd5836cf640 4 set_mon_vals no callback set 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd573fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd573fff640 10 monclient: dump: 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:52.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd5836cf640 10 monclient: shutdown 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd5836cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd5836cf640 1 -- 172.21.15.63:0/328580401 >> v1:172.21.15.63:6789/0 conn(0x7fd57c13f710 legacy=0x7fd57c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd5836cf640 1 -- 172.21.15.63:0/328580401 shutdown_connections 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.964+0000 7fd5836cf640 1 -- 172.21.15.63:0/328580401 >> 172.21.15.63:0/328580401 conn(0x7fd57c0b7ea0 msgr2=0x7fd57c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.965+0000 7fd5836cf640 1 -- 172.21.15.63:0/328580401 shutdown_connections 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.965+0000 7fd5836cf640 1 -- 172.21.15.63:0/328580401 wait complete. 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.966+0000 7fd5836cf640 10 monclient: build_initial_monmap 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.966+0000 7fd5836cf640 10 monclient: monmap: 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:22:52.974 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.966+0000 7fd5836cf640 1 librados: starting msgr at 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.966+0000 7fd5836cf640 1 librados: starting objecter 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.966+0000 7fd5836cf640 1 Processor -- start 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 1 -- start start 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 1 librados: setting wanted keys 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 1 librados: calling monclient init 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 10 monclient: init 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 10 monclient: _reopen_session rank -1 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd57c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:22:52.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd57c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:22:52.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd57c1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:22:52.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c1390e0 con 0x7fd57c13f710 2024-03-25T14:22:52.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c1db0a0 con 0x7fd57c1430c0 2024-03-25T14:22:52.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c137580 con 0x7fd57c0a98f0 2024-03-25T14:22:52.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.967+0000 7fd5836cf640 10 monclient(hunting): _renew_subs 2024-03-25T14:22:52.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.968+0000 7fd5836cf640 10 monclient(hunting): authenticate will time out at 1536.112530s 2024-03-25T14:22:52.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.984+0000 7fd581c6c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd57c1430c0 0x7fd57c133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:52716/0 (socket says 172.21.15.63:52716) 2024-03-25T14:22:52.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.984+0000 7fd581c6c640 1 -- 172.21.15.63:0/2691586370 learned_addr learned my addr 172.21.15.63:0/2691586370 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:22:52.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.985+0000 7fd571ffb640 10 client.?.objecter ms_handle_connect 0x7fd57c1430c0 2024-03-25T14:22:52.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.985+0000 7fd571ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:52.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.986+0000 7fd571ffb640 10 client.?.objecter ms_handle_connect 0x7fd57c0a98f0 2024-03-25T14:22:52.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.986+0000 7fd571ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:52.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.994+0000 7fd571ffb640 10 client.?.objecter ms_handle_connect 0x7fd57c13f710 2024-03-25T14:22:52.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.994+0000 7fd571ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:22:53.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.999+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681414006 0 0) 0x7fd57c1db0a0 con 0x7fd57c1430c0 2024-03-25T14:22:53.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.999+0000 7fd571ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:22:53.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.999+0000 7fd571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:53.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.999+0000 7fd571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:53.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.999+0000 7fd571ffb640 10 monclient(hunting): my global_id is 4366 2024-03-25T14:22:53.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:52.999+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd560004eb0 con 0x7fd57c1430c0 2024-03-25T14:22:53.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.002+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158163410 0 0) 0x7fd57c137580 con 0x7fd57c0a98f0 2024-03-25T14:22:53.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.002+0000 7fd571ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:22:53.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.002+0000 7fd571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:53.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.002+0000 7fd571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:53.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.002+0000 7fd571ffb640 10 monclient(hunting): my global_id is 4346 2024-03-25T14:22:53.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.002+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd57c1db0a0 con 0x7fd57c0a98f0 2024-03-25T14:22:53.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.011+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008327880 0 0) 0x7fd57c1390e0 con 0x7fd57c13f710 2024-03-25T14:22:53.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.011+0000 7fd571ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:22:53.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.011+0000 7fd571ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:22:53.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.011+0000 7fd571ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:22:53.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.011+0000 7fd571ffb640 10 monclient(hunting): my global_id is 4356 2024-03-25T14:22:53.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.011+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd57c137580 con 0x7fd57c13f710 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.014+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 898694759 0 0) 0x7fd560004eb0 con 0x7fd57c1430c0 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.014+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd57c1390e0 con 0x7fd57c1430c0 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.014+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd574001c30 con 0x7fd57c1430c0 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.014+0000 7fd571ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.014+0000 7fd571ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.014+0000 7fd571ffb640 10 monclient(hunting): dump: 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:53.018 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:53.019 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:53.019 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:53.019 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:53.019 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:53.019 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:53.019 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:53.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.014+0000 7fd571ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.015+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2235840846 0 0) 0x7fd57c1db0a0 con 0x7fd57c0a98f0 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.016+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd560004eb0 con 0x7fd57c0a98f0 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.016+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd5780018b0 con 0x7fd57c0a98f0 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.016+0000 7fd571ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.016+0000 7fd571ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.016+0000 7fd571ffb640 10 monclient(hunting): dump: 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:53.020 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:53.021 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:53.021 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:53.021 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:53.021 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:53.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.023+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1711988038 0 0) 0x7fd57c1390e0 con 0x7fd57c1430c0 2024-03-25T14:22:53.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 >> v1:172.21.15.63:6790/0 conn(0x7fd57c0a98f0 legacy=0x7fd57c1da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 >> v1:172.21.15.63:6789/0 conn(0x7fd57c13f710 legacy=0x7fd57c1369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 1 monclient: found mon.b 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c1370b0 con 0x7fd57c1430c0 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 10 monclient: _finish_auth 0 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd571ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 5 monclient: authenticate success, global_id 4366 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 10 monclient: _renew_subs 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c1337a0 con 0x7fd57c1430c0 2024-03-25T14:22:53.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:22:53.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 10 monclient: _renew_subs 2024-03-25T14:22:53.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:53.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c0b7350 con 0x7fd57c1430c0 2024-03-25T14:22:53.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.024+0000 7fd5836cf640 1 librados: init done 2024-03-25T14:22:53.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.025+0000 7fd5836cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:53.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.025+0000 7fd5836cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:53.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.025+0000 7fd5836cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:53.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.025+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c0627b0 con 0x7fd57c1430c0 2024-03-25T14:22:53.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd574003990 con 0x7fd57c1430c0 2024-03-25T14:22:53.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd571ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:22:53.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd574003da0 con 0x7fd57c1430c0 2024-03-25T14:22:53.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd571ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd571ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd571ffb640 10 monclient: dump: 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:22:53.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd5737fe640 4 set_mon_vals no callback set 2024-03-25T14:22:53.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.309+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd574003fc0 con 0x7fd57c1430c0 2024-03-25T14:22:53.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.310+0000 7fd58146b640 10 monclient: get_auth_request con 0x7fd56006df70 auth_method 0 2024-03-25T14:22:53.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.310+0000 7fd571ffb640 10 client.4366.objecter ms_handle_connect 0x7fd56006df70 2024-03-25T14:22:53.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.315+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fd574054a50 con 0x7fd57c1430c0 2024-03-25T14:22:53.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.315+0000 7fd571ffb640 10 client.4366.objecter ms_dispatch 0x7fd57c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:22:53.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.315+0000 7fd571ffb640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:22:53.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.315+0000 7fd571ffb640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:22:53.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.315+0000 7fd571ffb640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T14:22:53.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.316+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd574050560 con 0x7fd57c1430c0 2024-03-25T14:22:53.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.316+0000 7fd571ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:22:53.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.316+0000 7fd571ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:22:53.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.571+0000 7fd5836cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:53.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.572+0000 7fd5836cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:53.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.572+0000 7fd5836cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:22:53.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.572+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd57c133930 con 0x7fd57c1430c0 2024-03-25T14:22:53.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.617+0000 7fd571ffb640 1 -- 172.21.15.63:0/2691586370 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fd574052490 con 0x7fd57c1430c0 2024-03-25T14:22:53.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.617+0000 7fd571ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:22:53.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.617+0000 7fd571ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:22:53.621 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:22:53.621 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:22:53.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 10 librados: watch_flush enter 2024-03-25T14:22:53.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 10 librados: watch_flush exit 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 10 client.4366.objecter successfully canceled tick 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd56006df70 legacy=0x7fd560070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 10 monclient: shutdown 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 >> v1:172.21.15.153:6789/0 conn(0x7fd57c1430c0 legacy=0x7fd57c133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 shutdown_connections 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.620+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 >> 172.21.15.63:0/2691586370 conn(0x7fd57c0b7ea0 msgr2=0x7fd57c13ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.621+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 shutdown_connections 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.621+0000 7fd5836cf640 1 -- 172.21.15.63:0/2691586370 wait complete. 2024-03-25T14:22:53.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:22:53.621+0000 7fd5836cf640 1 librados: shutdown 2024-03-25T14:22:53.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:00.442 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.554+0000 7fb10ce38640 10 monclient: get_monmap_and_config 2024-03-25T14:23:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.554+0000 7fb10ce38640 10 monclient: build_initial_monmap 2024-03-25T14:23:00.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.554+0000 7fb10ce38640 10 monclient: monmap: 2024-03-25T14:23:00.558 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:00.555664+0000 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:00.555664+0000 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.555+0000 7fb10ce38640 1 Processor -- start 2024-03-25T14:23:00.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.555+0000 7fb10ce38640 1 -- start start 2024-03-25T14:23:00.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.555+0000 7fb10ce38640 10 monclient: init 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 10 monclient(hunting): picked mon.noname-b con 0x7fb1080b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 10 monclient(hunting): picked mon.noname-c con 0x7fb1080b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 10 monclient(hunting): picked mon.noname-a con 0x7fb10814be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080af940 con 0x7fb10814be90 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080b0a70 con 0x7fb1080b1730 2024-03-25T14:23:00.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080b0d50 con 0x7fb1080b4d60 2024-03-25T14:23:00.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:00.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.556+0000 7fb10ce38640 10 monclient(hunting): authenticate will time out at 1543.701065s 2024-03-25T14:23:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.560+0000 7fb106d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb10814be90 0x7fb10814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47364/0 (socket says 172.21.15.63:47364) 2024-03-25T14:23:00.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.560+0000 7fb106d9d640 1 -- 172.21.15.63:0/527829021 learned_addr learned my addr 172.21.15.63:0/527829021 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:00.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.579+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372583968 0 0) 0x7fb1080af940 con 0x7fb10814be90 2024-03-25T14:23:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.579+0000 7fb10559a640 10 monclient(hunting): _negotiate 2024-03-25T14:23:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.579+0000 7fb10559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.579+0000 7fb10559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.579+0000 7fb10559a640 10 monclient(hunting): my global_id is 4359 2024-03-25T14:23:00.583 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.579+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f0003520 con 0x7fb10814be90 2024-03-25T14:23:00.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432488155 0 0) 0x7fb1080b0d50 con 0x7fb1080b4d60 2024-03-25T14:23:00.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): _negotiate 2024-03-25T14:23:00.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): my global_id is 4349 2024-03-25T14:23:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1080af940 con 0x7fb1080b4d60 2024-03-25T14:23:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193707586 0 0) 0x7fb1080b0a70 con 0x7fb1080b1730 2024-03-25T14:23:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): _negotiate 2024-03-25T14:23:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:00.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 10 monclient(hunting): my global_id is 4369 2024-03-25T14:23:00.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.587+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1080b0d50 con 0x7fb1080b1730 2024-03-25T14:23:00.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4205347459 0 0) 0x7fb1080b0d50 con 0x7fb1080b1730 2024-03-25T14:23:00.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 >> v1:172.21.15.63:6790/0 conn(0x7fb1080b4d60 legacy=0x7fb10813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 >> v1:172.21.15.63:6789/0 conn(0x7fb10814be90 legacy=0x7fb10814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 10 monclient: _finish_hunting 0 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 1 monclient: found mon.noname-c 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080b1030 con 0x7fb1080b1730 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 10 monclient: _finish_auth 0 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 10 monclient: _check_auth_tickets 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb0fc0015b0 con 0x7fb1080b1730 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10ce38640 5 monclient: authenticate success, global_id 4369 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10ce38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:00.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10559a640 10 monclient: dump: 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.595+0000 7fb10ce38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0fc001790 con 0x7fb1080b1730 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10559a640 1 -- 172.21.15.63:0/527829021 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb0fc001be0 con 0x7fb1080b1730 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10559a640 10 monclient: dump: 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.607 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10ce38640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:00.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.603+0000 7fb10ce38640 4 set_mon_vals no callback set 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.604+0000 7fb10ce38640 10 monclient: shutdown 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.604+0000 7fb10ce38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.604+0000 7fb10ce38640 1 -- 172.21.15.63:0/527829021 >> v1:172.21.15.153:6789/0 conn(0x7fb1080b1730 legacy=0x7fb1080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.604+0000 7fb10ce38640 1 -- 172.21.15.63:0/527829021 shutdown_connections 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.604+0000 7fb10ce38640 1 -- 172.21.15.63:0/527829021 >> 172.21.15.63:0/527829021 conn(0x7fb1080617a0 msgr2=0x7fb108061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.605+0000 7fb10ce38640 1 -- 172.21.15.63:0/527829021 shutdown_connections 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.605+0000 7fb10ce38640 1 -- 172.21.15.63:0/527829021 wait complete. 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.605+0000 7fb10ce38640 10 monclient: build_initial_monmap 2024-03-25T14:23:00.610 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.605+0000 7fb10ce38640 10 monclient: monmap: 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 librados: starting msgr at 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 librados: starting objecter 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 Processor -- start 2024-03-25T14:23:00.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 -- start start 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 librados: setting wanted keys 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 librados: calling monclient init 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient: init 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient(hunting): picked mon.noname-c con 0x7fb1080b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient(hunting): picked mon.noname-a con 0x7fb1080b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient(hunting): picked mon.noname-b con 0x7fb10814be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb108137b30 con 0x7fb1080b4d60 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb108137d60 con 0x7fb10814be90 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb108136360 con 0x7fb1080b1730 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:00.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.606+0000 7fb10ce38640 10 monclient(hunting): authenticate will time out at 1543.751464s 2024-03-25T14:23:00.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.613+0000 7fb105d9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb1080b4d60 0x7fb10812fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47396/0 (socket says 172.21.15.63:47396) 2024-03-25T14:23:00.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.613+0000 7fb105d9b640 1 -- 172.21.15.63:0/2453530405 learned_addr learned my addr 172.21.15.63:0/2453530405 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:00.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.614+0000 7fb0f6ffd640 10 client.?.objecter ms_handle_connect 0x7fb1080b4d60 2024-03-25T14:23:00.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.614+0000 7fb0f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:00.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.618+0000 7fb0f6ffd640 10 client.?.objecter ms_handle_connect 0x7fb1080b1730 2024-03-25T14:23:00.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.618+0000 7fb0f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:00.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.621+0000 7fb0f6ffd640 10 client.?.objecter ms_handle_connect 0x7fb10814be90 2024-03-25T14:23:00.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.621+0000 7fb0f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:00.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.625+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806040898 0 0) 0x7fb108137b30 con 0x7fb1080b4d60 2024-03-25T14:23:00.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.625+0000 7fb0f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:00.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.625+0000 7fb0f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:00.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.625+0000 7fb0f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:00.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.625+0000 7fb0f6ffd640 10 monclient(hunting): my global_id is 4362 2024-03-25T14:23:00.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.625+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e8004eb0 con 0x7fb1080b4d60 2024-03-25T14:23:00.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.626+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886262663 0 0) 0x7fb108136360 con 0x7fb1080b1730 2024-03-25T14:23:00.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.626+0000 7fb0f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:00.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.626+0000 7fb0f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:00.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.626+0000 7fb0f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:00.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.626+0000 7fb0f6ffd640 10 monclient(hunting): my global_id is 4352 2024-03-25T14:23:00.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.626+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb108137b30 con 0x7fb1080b1730 2024-03-25T14:23:00.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.634+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194876240 0 0) 0x7fb108137d60 con 0x7fb10814be90 2024-03-25T14:23:00.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.634+0000 7fb0f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:00.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.634+0000 7fb0f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:00.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.634+0000 7fb0f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:00.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.634+0000 7fb0f6ffd640 10 monclient(hunting): my global_id is 4372 2024-03-25T14:23:00.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.634+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb108136360 con 0x7fb10814be90 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.638+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000027387 0 0) 0x7fb108137b30 con 0x7fb1080b1730 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.638+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb108137d60 con 0x7fb1080b1730 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.638+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb0f8001930 con 0x7fb1080b1730 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.638+0000 7fb0f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.638+0000 7fb0f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.638+0000 7fb0f6ffd640 10 monclient(hunting): dump: 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.642 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:00.643 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:00.643 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:00.643 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:00.643 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:00.643 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.638+0000 7fb0f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:00.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.642+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3489933235 0 0) 0x7fb0e8004eb0 con 0x7fb1080b4d60 2024-03-25T14:23:00.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.642+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb108137b30 con 0x7fb1080b4d60 2024-03-25T14:23:00.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.642+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb0fc001630 con 0x7fb1080b4d60 2024-03-25T14:23:00.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.642+0000 7fb0f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:00.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.642+0000 7fb0f6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:00.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.642+0000 7fb0f6ffd640 10 monclient(hunting): dump: 2024-03-25T14:23:00.646 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:00.647 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 93562817 0 0) 0x7fb108136360 con 0x7fb10814be90 2024-03-25T14:23:00.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e8004eb0 con 0x7fb10814be90 2024-03-25T14:23:00.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb100001a70 con 0x7fb10814be90 2024-03-25T14:23:00.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:00.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:00.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 10 monclient(hunting): dump: 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 660265547 0 0) 0x7fb108137d60 con 0x7fb1080b1730 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 >> v1:172.21.15.153:6789/0 conn(0x7fb10814be90 legacy=0x7fb1081d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 >> v1:172.21.15.63:6789/0 conn(0x7fb1080b4d60 legacy=0x7fb10812fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:00.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 monclient: found mon.c 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb108130500 con 0x7fb1080b1730 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 10 monclient: _finish_auth 0 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb0f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 5 monclient: authenticate success, global_id 4352 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 10 monclient: _renew_subs 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1081307b0 con 0x7fb1080b1730 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 10 monclient: _renew_subs 2024-03-25T14:23:00.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:00.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1081325d0 con 0x7fb1080b1730 2024-03-25T14:23:00.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.655+0000 7fb10ce38640 1 librados: init done 2024-03-25T14:23:00.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.656+0000 7fb10ce38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:00.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.656+0000 7fb10ce38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:00.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.656+0000 7fb10ce38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:00.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.656+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1081401f0 con 0x7fb1080b1730 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.972+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0f8001d60 con 0x7fb1080b1730 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.972+0000 7fb0f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.972+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb0f80037b0 con 0x7fb1080b1730 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.972+0000 7fb0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.972+0000 7fb0f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.972+0000 7fb0f6ffd640 10 monclient: dump: 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:00.976 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:00.977 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:00.977 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:00.977 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:00.977 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:00.977 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:00.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.972+0000 7fb104d99640 4 set_mon_vals no callback set 2024-03-25T14:23:00.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.973+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fb0f8019780 con 0x7fb1080b1730 2024-03-25T14:23:00.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.973+0000 7fb105d9b640 10 monclient: get_auth_request con 0x7fb0e806e0f0 auth_method 0 2024-03-25T14:23:00.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.974+0000 7fb0f6ffd640 10 client.4352.objecter ms_handle_connect 0x7fb0e806e0f0 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fb0f80534e0 con 0x7fb1080b1730 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 10 client.4352.objecter ms_dispatch 0x7fb108000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0f804ee20 con 0x7fb1080b1730 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:00.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:00.980+0000 7fb0f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:01.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.226+0000 7fb10ce38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:01.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.226+0000 7fb10ce38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:01.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.226+0000 7fb10ce38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:01.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.226+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1081364f0 con 0x7fb1080b1730 2024-03-25T14:23:01.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.379+0000 7fb0f6ffd640 1 -- 172.21.15.63:0/2453530405 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fb0f8050f20 con 0x7fb1080b1730 2024-03-25T14:23:01.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.379+0000 7fb0f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:01.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.379+0000 7fb0f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:01.383 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:01.383 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 10 librados: watch_flush enter 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 10 librados: watch_flush exit 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 10 client.4352.objecter successfully canceled tick 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 >> v1:172.21.15.153:6800/1688255552 conn(0x7fb0e806e0f0 legacy=0x7fb0e80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 10 monclient: shutdown 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:01.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 >> v1:172.21.15.63:6790/0 conn(0x7fb1080b1730 legacy=0x7fb108135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:01.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 shutdown_connections 2024-03-25T14:23:01.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 >> 172.21.15.63:0/2453530405 conn(0x7fb1080617a0 msgr2=0x7fb1080b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:01.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 shutdown_connections 2024-03-25T14:23:01.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 1 -- 172.21.15.63:0/2453530405 wait complete. 2024-03-25T14:23:01.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:01.382+0000 7fb10ce38640 1 librados: shutdown 2024-03-25T14:23:01.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:06.331 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:44.844 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.339 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:44.852 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.382 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:03:44.895 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.390 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:44.850 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.393 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:44.906 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.398 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:44.858 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.404 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:44.917 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.424 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:44.885 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:06.430 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:03:44.943 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:08.203 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:08.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.316+0000 7f29ddb5d640 10 monclient: get_monmap_and_config 2024-03-25T14:23:08.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.316+0000 7f29ddb5d640 10 monclient: build_initial_monmap 2024-03-25T14:23:08.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.316+0000 7f29ddb5d640 10 monclient: monmap: 2024-03-25T14:23:08.318 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:08.316332+0000 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:08.316332+0000 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:08.319 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.317+0000 7f29ddb5d640 1 Processor -- start 2024-03-25T14:23:08.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.317+0000 7f29ddb5d640 1 -- start start 2024-03-25T14:23:08.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.317+0000 7f29ddb5d640 10 monclient: init 2024-03-25T14:23:08.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:08.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:23:08.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f29d80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:08.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f29d813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:08.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f29d81430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:08.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d8142520 con 0x7f29d80a98f0 2024-03-25T14:23:08.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d80b28a0 con 0x7f29d813f710 2024-03-25T14:23:08.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d80b2a30 con 0x7f29d81430c0 2024-03-25T14:23:08.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:08.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.318+0000 7f29ddb5d640 10 monclient(hunting): authenticate will time out at 1551.462090s 2024-03-25T14:23:08.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.323+0000 7f29d7fff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f29d81430c0 0x7f29d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:59978/0 (socket says 172.21.15.63:59978) 2024-03-25T14:23:08.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.323+0000 7f29d7fff640 1 -- 172.21.15.63:0/3255340387 learned_addr learned my addr 172.21.15.63:0/3255340387 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:08.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.337+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069431147 0 0) 0x7f29d80b2a30 con 0x7f29d81430c0 2024-03-25T14:23:08.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.337+0000 7f29d67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:23:08.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.337+0000 7f29d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:08.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.337+0000 7f29d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:08.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.337+0000 7f29d67fc640 10 monclient(hunting): my global_id is 4355 2024-03-25T14:23:08.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.337+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c0003520 con 0x7f29d81430c0 2024-03-25T14:23:08.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.340+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320633812 0 0) 0x7f29d80b28a0 con 0x7f29d813f710 2024-03-25T14:23:08.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.341+0000 7f29d67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:23:08.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.341+0000 7f29d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:08.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.341+0000 7f29d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:08.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.341+0000 7f29d67fc640 10 monclient(hunting): my global_id is 4375 2024-03-25T14:23:08.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.341+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d80b2a30 con 0x7f29d813f710 2024-03-25T14:23:08.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626362106 0 0) 0x7f29c0003520 con 0x7f29d81430c0 2024-03-25T14:23:08.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 >> v1:172.21.15.153:6789/0 conn(0x7f29d813f710 legacy=0x7f29d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:08.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 >> v1:172.21.15.63:6789/0 conn(0x7f29d80a98f0 legacy=0x7f29d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:08.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:23:08.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 1 monclient: found mon.noname-b 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80b44c0 con 0x7f29d81430c0 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 10 monclient: _finish_auth 0 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f29d0001a10 con 0x7f29d81430c0 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29d67fc640 10 monclient: dump: 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.356 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.352+0000 7f29ddb5d640 5 monclient: authenticate success, global_id 4355 2024-03-25T14:23:08.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.353+0000 7f29ddb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29d0001b70 con 0x7f29d81430c0 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29d67fc640 1 -- 172.21.15.63:0/3255340387 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f29d0001f80 con 0x7f29d81430c0 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29ddb5d640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29ddb5d640 4 set_mon_vals no callback set 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.365+0000 7f29d67fc640 10 monclient: dump: 2024-03-25T14:23:08.368 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:08.369 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.366+0000 7f29ddb5d640 10 monclient: shutdown 2024-03-25T14:23:08.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.366+0000 7f29ddb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:08.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.366+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3255340387 >> v1:172.21.15.63:6790/0 conn(0x7f29d81430c0 legacy=0x7f29d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:08.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.366+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3255340387 shutdown_connections 2024-03-25T14:23:08.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.366+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3255340387 >> 172.21.15.63:0/3255340387 conn(0x7f29d80b7ea0 msgr2=0x7f29d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.366+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3255340387 shutdown_connections 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.366+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3255340387 wait complete. 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.367+0000 7f29ddb5d640 10 monclient: build_initial_monmap 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.367+0000 7f29ddb5d640 10 monclient: monmap: 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 1 librados: starting msgr at 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 1 librados: starting objecter 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 1 Processor -- start 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 1 -- start start 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 1 librados: setting wanted keys 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 1 librados: calling monclient init 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 10 monclient: init 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f29d80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.368+0000 7f29ddb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f29d813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.369+0000 7f29ddb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f29d81430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.369+0000 7f29ddb5d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d81396e0 con 0x7f29d81430c0 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.369+0000 7f29ddb5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d8138530 con 0x7f29d813f710 2024-03-25T14:23:08.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.369+0000 7f29ddb5d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d81399c0 con 0x7f29d80a98f0 2024-03-25T14:23:08.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.369+0000 7f29ddb5d640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:08.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.369+0000 7f29ddb5d640 10 monclient(hunting): authenticate will time out at 1551.512665s 2024-03-25T14:23:08.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.373+0000 7f29d7fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f29d81430c0 0x7f29d81e1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51972/0 (socket says 172.21.15.63:51972) 2024-03-25T14:23:08.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.373+0000 7f29d7fff640 1 -- 172.21.15.63:0/3665374901 learned_addr learned my addr 172.21.15.63:0/3665374901 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:08.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.376+0000 7f29bffff640 10 client.?.objecter ms_handle_connect 0x7f29d813f710 2024-03-25T14:23:08.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.376+0000 7f29bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:08.381 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.378+0000 7f29bffff640 10 client.?.objecter ms_handle_connect 0x7f29d80a98f0 2024-03-25T14:23:08.381 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.378+0000 7f29bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:08.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.384+0000 7f29bffff640 10 client.?.objecter ms_handle_connect 0x7f29d81430c0 2024-03-25T14:23:08.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.384+0000 7f29bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:08.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.391+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076625017 0 0) 0x7f29d8138530 con 0x7f29d813f710 2024-03-25T14:23:08.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.391+0000 7f29bffff640 10 monclient(hunting): _negotiate 2024-03-25T14:23:08.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.391+0000 7f29bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:08.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.391+0000 7f29bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:08.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.391+0000 7f29bffff640 10 monclient(hunting): my global_id is 4378 2024-03-25T14:23:08.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.391+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c4004eb0 con 0x7f29d813f710 2024-03-25T14:23:08.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.399+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108333925 0 0) 0x7f29d81399c0 con 0x7f29d80a98f0 2024-03-25T14:23:08.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.399+0000 7f29bffff640 10 monclient(hunting): _negotiate 2024-03-25T14:23:08.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.399+0000 7f29bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:08.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.399+0000 7f29bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:08.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.399+0000 7f29bffff640 10 monclient(hunting): my global_id is 4358 2024-03-25T14:23:08.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.399+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d8138530 con 0x7f29d80a98f0 2024-03-25T14:23:08.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.402+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032525768 0 0) 0x7f29d81396e0 con 0x7f29d81430c0 2024-03-25T14:23:08.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.402+0000 7f29bffff640 10 monclient(hunting): _negotiate 2024-03-25T14:23:08.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.402+0000 7f29bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:08.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.402+0000 7f29bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:08.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.402+0000 7f29bffff640 10 monclient(hunting): my global_id is 4368 2024-03-25T14:23:08.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.402+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d81399c0 con 0x7f29d81430c0 2024-03-25T14:23:08.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.407+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310685172 0 0) 0x7f29c4004eb0 con 0x7f29d813f710 2024-03-25T14:23:08.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.407+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29d81396e0 con 0x7f29d813f710 2024-03-25T14:23:08.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.407+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f29cc001930 con 0x7f29d813f710 2024-03-25T14:23:08.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.407+0000 7f29bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:08.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.407+0000 7f29bffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:23:08.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.407+0000 7f29bffff640 10 monclient(hunting): dump: 2024-03-25T14:23:08.409 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.407+0000 7f29bffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.422+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3605277090 0 0) 0x7f29d81399c0 con 0x7f29d81430c0 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.422+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29c4004eb0 con 0x7f29d81430c0 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.422+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f29d0001e80 con 0x7f29d81430c0 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.422+0000 7f29bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.422+0000 7f29bffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.422+0000 7f29bffff640 10 monclient(hunting): dump: 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:08.425 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.426 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.426 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:08.426 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:08.426 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:08.426 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:08.426 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:08.426 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.425+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2198879705 0 0) 0x7f29d8138530 con 0x7f29d80a98f0 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.425+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29d81399c0 con 0x7f29d80a98f0 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.425+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f29c80018a0 con 0x7f29d80a98f0 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.425+0000 7f29bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.425+0000 7f29bffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.425+0000 7f29bffff640 10 monclient(hunting): dump: 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:08.428 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:08.429 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:08.429 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:08.429 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:08.429 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1692709322 0 0) 0x7f29d81396e0 con 0x7f29d813f710 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 >> v1:172.21.15.63:6790/0 conn(0x7f29d80a98f0 legacy=0x7f29d8137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 >> v1:172.21.15.63:6789/0 conn(0x7f29d81430c0 legacy=0x7f29d81e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 10 monclient: _finish_hunting 0 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 1 monclient: found mon.b 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8138a50 con 0x7f29d813f710 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 10 monclient: _finish_auth 0 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 10 monclient: _check_auth_tickets 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 5 monclient: authenticate success, global_id 4378 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 10 monclient: _renew_subs 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:08.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d81380a0 con 0x7f29d813f710 2024-03-25T14:23:08.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:08.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 10 monclient: _renew_subs 2024-03-25T14:23:08.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:08.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d81db140 con 0x7f29d813f710 2024-03-25T14:23:08.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.428+0000 7f29ddb5d640 1 librados: init done 2024-03-25T14:23:08.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.429+0000 7f29ddb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:08.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.429+0000 7f29ddb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:08.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.429+0000 7f29ddb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:08.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.429+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d81383a0 con 0x7f29d813f710 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.728+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29cc003610 con 0x7f29d813f710 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.728+0000 7f29bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.728+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f29cc003920 con 0x7f29d813f710 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.728+0000 7f29bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.728+0000 7f29bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.728+0000 7f29bffff640 10 monclient: dump: 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:08.731 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:08.732 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:08.732 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:08.732 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:08.732 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:08.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.728+0000 7f29d5ffb640 4 set_mon_vals no callback set 2024-03-25T14:23:08.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.729+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f29cc019970 con 0x7f29d813f710 2024-03-25T14:23:08.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.730+0000 7f29d77fe640 10 monclient: get_auth_request con 0x7f29c406e0f0 auth_method 0 2024-03-25T14:23:08.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.730+0000 7f29bffff640 10 client.4378.objecter ms_handle_connect 0x7f29c406e0f0 2024-03-25T14:23:08.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.733+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f29cc053610 con 0x7f29d813f710 2024-03-25T14:23:08.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.733+0000 7f29bffff640 10 client.4378.objecter ms_dispatch 0x7f29d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:08.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.733+0000 7f29bffff640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:08.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.733+0000 7f29bffff640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:08.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.733+0000 7f29bffff640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T14:23:08.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.734+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29cc04ef50 con 0x7f29d813f710 2024-03-25T14:23:08.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.734+0000 7f29bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:08.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.734+0000 7f29bffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:08.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.992+0000 7f29ddb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:08.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.992+0000 7f29ddb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:08.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.992+0000 7f29ddb5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:08.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:08.992+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29d80627b0 con 0x7f29d813f710 2024-03-25T14:23:09.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.014+0000 7f29bffff640 1 -- 172.21.15.63:0/3665374901 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f29cc051050 con 0x7f29d813f710 2024-03-25T14:23:09.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.014+0000 7f29bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:09.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.014+0000 7f29bffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:09.017 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:09.017 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:09.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.016+0000 7f29ddb5d640 10 librados: watch_flush enter 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.016+0000 7f29ddb5d640 10 librados: watch_flush exit 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.016+0000 7f29ddb5d640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.016+0000 7f29ddb5d640 10 client.4378.objecter successfully canceled tick 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 >> v1:172.21.15.153:6800/1688255552 conn(0x7f29c406e0f0 legacy=0x7f29c40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 10 monclient: shutdown 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 >> v1:172.21.15.153:6789/0 conn(0x7f29d813f710 legacy=0x7f29d81da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 shutdown_connections 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 >> 172.21.15.63:0/3665374901 conn(0x7f29d80b7ea0 msgr2=0x7f29d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 shutdown_connections 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 1 -- 172.21.15.63:0/3665374901 wait complete. 2024-03-25T14:23:09.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:09.017+0000 7f29ddb5d640 1 librados: shutdown 2024-03-25T14:23:09.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:15.837 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:15.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.954+0000 7f361960e640 10 monclient: get_monmap_and_config 2024-03-25T14:23:15.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.954+0000 7f361960e640 10 monclient: build_initial_monmap 2024-03-25T14:23:15.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.954+0000 7f361960e640 10 monclient: monmap: 2024-03-25T14:23:15.957 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:15.955213+0000 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:15.955213+0000 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:15.958 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:15.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.954+0000 7f361960e640 1 Processor -- start 2024-03-25T14:23:15.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.954+0000 7f361960e640 1 -- start start 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.954+0000 7f361960e640 10 monclient: init 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 10 monclient(hunting): picked mon.noname-c con 0x7f36140a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 10 monclient(hunting): picked mon.noname-a con 0x7f361413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 10 monclient(hunting): picked mon.noname-b con 0x7f36141430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3614142520 con 0x7f361413f710 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36140b28a0 con 0x7f36140a98f0 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36140b2a30 con 0x7f36141430c0 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:15.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.955+0000 7f361960e640 10 monclient(hunting): authenticate will time out at 1559.100313s 2024-03-25T14:23:15.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.970+0000 7f3612ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f36140a98f0 0x7f36140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:41218/0 (socket says 172.21.15.63:41218) 2024-03-25T14:23:15.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.970+0000 7f3612ffd640 1 -- 172.21.15.63:0/3536713208 learned_addr learned my addr 172.21.15.63:0/3536713208 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:15.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.982+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606079462 0 0) 0x7f36140b28a0 con 0x7f36140a98f0 2024-03-25T14:23:15.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.982+0000 7f3611ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:23:15.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.982+0000 7f3611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:15.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.982+0000 7f3611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:15.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.982+0000 7f3611ffb640 10 monclient(hunting): my global_id is 4381 2024-03-25T14:23:15.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.982+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35fc003520 con 0x7f36140a98f0 2024-03-25T14:23:15.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.995+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717210708 0 0) 0x7f36140b2a30 con 0x7f36141430c0 2024-03-25T14:23:15.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.995+0000 7f3611ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:23:15.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.995+0000 7f3611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:15.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.995+0000 7f3611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:15.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.995+0000 7f3611ffb640 10 monclient(hunting): my global_id is 4361 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.995+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36140b28a0 con 0x7f36141430c0 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1886470797 0 0) 0x7f35fc003520 con 0x7f36140a98f0 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 >> v1:172.21.15.63:6790/0 conn(0x7f36141430c0 legacy=0x7f361414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 >> v1:172.21.15.63:6789/0 conn(0x7f361413f710 legacy=0x7f36141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 1 monclient: found mon.noname-c 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36140b44c0 con 0x7f36140a98f0 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 10 monclient: _finish_auth 0 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3604001690 con 0x7f36140a98f0 2024-03-25T14:23:16.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f3611ffb640 10 monclient: dump: 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:16.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f361960e640 5 monclient: authenticate success, global_id 4381 2024-03-25T14:23:16.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:15.996+0000 7f361960e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36040017f0 con 0x7f36140a98f0 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f3611ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f3611ffb640 1 -- 172.21.15.63:0/3536713208 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3604001c00 con 0x7f36140a98f0 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f3611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f3611ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f3611ffb640 10 monclient: dump: 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:16.020 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:16.021 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:16.021 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:16.021 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:16.021 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:16.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f361960e640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:16.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.016+0000 7f361960e640 4 set_mon_vals no callback set 2024-03-25T14:23:16.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.017+0000 7f361960e640 10 monclient: shutdown 2024-03-25T14:23:16.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.017+0000 7f361960e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:16.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.017+0000 7f361960e640 1 -- 172.21.15.63:0/3536713208 >> v1:172.21.15.153:6789/0 conn(0x7f36140a98f0 legacy=0x7f36140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:16.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.017+0000 7f361960e640 1 -- 172.21.15.63:0/3536713208 shutdown_connections 2024-03-25T14:23:16.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.017+0000 7f361960e640 1 -- 172.21.15.63:0/3536713208 >> 172.21.15.63:0/3536713208 conn(0x7f36140b7ea0 msgr2=0x7f36140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:16.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.017+0000 7f361960e640 1 -- 172.21.15.63:0/3536713208 shutdown_connections 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.017+0000 7f361960e640 1 -- 172.21.15.63:0/3536713208 wait complete. 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.018+0000 7f361960e640 10 monclient: build_initial_monmap 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.018+0000 7f361960e640 10 monclient: monmap: 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.018+0000 7f361960e640 1 librados: starting msgr at 2024-03-25T14:23:16.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.018+0000 7f361960e640 1 librados: starting objecter 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 1 Processor -- start 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 1 -- start start 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 1 librados: setting wanted keys 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 1 librados: calling monclient init 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 10 monclient: init 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 10 monclient(hunting): picked mon.noname-c con 0x7f36140a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 10 monclient(hunting): picked mon.noname-b con 0x7f361413f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:16.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 10 monclient(hunting): picked mon.noname-a con 0x7f36141430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:16.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3614070750 con 0x7f36141430c0 2024-03-25T14:23:16.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3614070a30 con 0x7f361413f710 2024-03-25T14:23:16.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361406f880 con 0x7f36140a98f0 2024-03-25T14:23:16.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.019+0000 7f361960e640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:16.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.020+0000 7f361960e640 10 monclient(hunting): authenticate will time out at 1559.164519s 2024-03-25T14:23:16.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.027+0000 7f36137fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f36141430c0 0x7f36141e67a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36038/0 (socket says 172.21.15.63:36038) 2024-03-25T14:23:16.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.027+0000 7f36137fe640 1 -- 172.21.15.63:0/1568975232 learned_addr learned my addr 172.21.15.63:0/1568975232 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:16.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.028+0000 7f35fb7fe640 10 client.?.objecter ms_handle_connect 0x7f36141430c0 2024-03-25T14:23:16.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.028+0000 7f35fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:16.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.033+0000 7f35fb7fe640 10 client.?.objecter ms_handle_connect 0x7f36140a98f0 2024-03-25T14:23:16.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.033+0000 7f35fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:16.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.036+0000 7f35fb7fe640 10 client.?.objecter ms_handle_connect 0x7f361413f710 2024-03-25T14:23:16.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.036+0000 7f35fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:16.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.043+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206922867 0 0) 0x7f3614070750 con 0x7f36141430c0 2024-03-25T14:23:16.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:23:16.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:16.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:16.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T14:23:16.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3600004eb0 con 0x7f36141430c0 2024-03-25T14:23:16.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613354735 0 0) 0x7f3614070a30 con 0x7f361413f710 2024-03-25T14:23:16.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:23:16.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:16.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:16.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T14:23:16.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.044+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3614070750 con 0x7f361413f710 2024-03-25T14:23:16.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.051+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083621289 0 0) 0x7f361406f880 con 0x7f36140a98f0 2024-03-25T14:23:16.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.051+0000 7f35fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:23:16.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.051+0000 7f35fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:16.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.051+0000 7f35fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:16.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.051+0000 7f35fb7fe640 10 monclient(hunting): my global_id is 4364 2024-03-25T14:23:16.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.051+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3614070a30 con 0x7f36140a98f0 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.053+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3866809775 0 0) 0x7f3614070750 con 0x7f361413f710 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.053+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f361406f880 con 0x7f361413f710 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.053+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f36080018a0 con 0x7f361413f710 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.053+0000 7f35fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.054+0000 7f35fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.054+0000 7f35fb7fe640 10 monclient(hunting): dump: 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:16.057 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:16.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.054+0000 7f35fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.063+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296647261 0 0) 0x7f3600004eb0 con 0x7f36141430c0 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.063+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3614070750 con 0x7f36141430c0 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.063+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f360c001d20 con 0x7f36141430c0 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.063+0000 7f35fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.063+0000 7f35fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.063+0000 7f35fb7fe640 10 monclient(hunting): dump: 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:16.067 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:16.068 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:16.068 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:16.068 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:16.068 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:16.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3762741988 0 0) 0x7f361406f880 con 0x7f361413f710 2024-03-25T14:23:16.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 >> v1:172.21.15.63:6790/0 conn(0x7f36140a98f0 legacy=0x7f361406f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:16.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 >> v1:172.21.15.63:6789/0 conn(0x7f36141430c0 legacy=0x7f36141e67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:16.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:23:16.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 1 monclient: found mon.b 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361406fa10 con 0x7f361413f710 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 10 monclient: _finish_auth 0 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f35fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f361960e640 5 monclient: authenticate success, global_id 4384 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f361960e640 10 monclient: _renew_subs 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f361960e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.071+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361406eb80 con 0x7f361413f710 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.072+0000 7f361960e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.072+0000 7f361960e640 10 monclient: _renew_subs 2024-03-25T14:23:16.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.072+0000 7f361960e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:16.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.072+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3614071030 con 0x7f361413f710 2024-03-25T14:23:16.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.072+0000 7f361960e640 1 librados: init done 2024-03-25T14:23:16.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.073+0000 7f361960e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:16.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.073+0000 7f361960e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:16.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.073+0000 7f361960e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:16.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.073+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36140627b0 con 0x7f361413f710 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.360+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3608001d50 con 0x7f361413f710 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.360+0000 7f35fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.360+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3608003790 con 0x7f361413f710 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.360+0000 7f35fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.360+0000 7f35fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.360+0000 7f35fb7fe640 10 monclient: dump: 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:16.364 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:16.365 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:16.365 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:16.365 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:16.365 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:16.365 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:16.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.360+0000 7f36117fa640 4 set_mon_vals no callback set 2024-03-25T14:23:16.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.361+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f36080039b0 con 0x7f361413f710 2024-03-25T14:23:16.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.362+0000 7f3612ffd640 10 monclient: get_auth_request con 0x7f360006df70 auth_method 0 2024-03-25T14:23:16.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.362+0000 7f35fb7fe640 10 client.4384.objecter ms_handle_connect 0x7f360006df70 2024-03-25T14:23:16.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f3608053570 con 0x7f361413f710 2024-03-25T14:23:16.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 10 client.4384.objecter ms_dispatch 0x7f3614000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:16.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:16.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:16.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T14:23:16.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f360804eeb0 con 0x7f361413f710 2024-03-25T14:23:16.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:16.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.366+0000 7f35fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:16.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.630+0000 7f361960e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:16.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.630+0000 7f361960e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:16.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.630+0000 7f361960e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:16.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.630+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f361413dcf0 con 0x7f361413f710 2024-03-25T14:23:16.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.652+0000 7f35fb7fe640 1 -- 172.21.15.63:0/1568975232 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f3608050fb0 con 0x7f361413f710 2024-03-25T14:23:16.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.652+0000 7f35fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:16.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.652+0000 7f35fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:16.656 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:16.657 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 10 librados: watch_flush enter 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 10 librados: watch_flush exit 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 10 client.4384.objecter successfully canceled tick 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 >> v1:172.21.15.153:6800/1688255552 conn(0x7f360006df70 legacy=0x7f3600070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 10 monclient: shutdown 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 >> v1:172.21.15.153:6789/0 conn(0x7f361413f710 legacy=0x7f36141dc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 shutdown_connections 2024-03-25T14:23:16.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 >> 172.21.15.63:0/1568975232 conn(0x7f36140b7ea0 msgr2=0x7f3614074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:16.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 shutdown_connections 2024-03-25T14:23:16.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 1 -- 172.21.15.63:0/1568975232 wait complete. 2024-03-25T14:23:16.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:16.655+0000 7f361960e640 1 librados: shutdown 2024-03-25T14:23:16.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:23.478 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:23.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.587+0000 7f3deb08f640 10 monclient: get_monmap_and_config 2024-03-25T14:23:23.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.587+0000 7f3deb08f640 10 monclient: build_initial_monmap 2024-03-25T14:23:23.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.587+0000 7f3deb08f640 10 monclient: monmap: 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:23.588189+0000 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:23.588189+0000 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:23.591 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:23.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.587+0000 7f3deb08f640 1 Processor -- start 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 1 -- start start 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient: init 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient(hunting): picked mon.noname-a con 0x7f3de40a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient(hunting): picked mon.noname-b con 0x7f3de413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient(hunting): picked mon.noname-c con 0x7f3de41430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de4142520 con 0x7f3de40a98f0 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40b28a0 con 0x7f3de41430c0 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40b2a30 con 0x7f3de413f710 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:23.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.588+0000 7f3deb08f640 10 monclient(hunting): authenticate will time out at 1566.733442s 2024-03-25T14:23:23.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.594+0000 7f3de962c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3de41430c0 0x7f3de414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36712/0 (socket says 172.21.15.63:36712) 2024-03-25T14:23:23.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.594+0000 7f3de962c640 1 -- 172.21.15.63:0/4102515317 learned_addr learned my addr 172.21.15.63:0/4102515317 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:23.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.607+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102835399 0 0) 0x7f3de40b2a30 con 0x7f3de413f710 2024-03-25T14:23:23.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.607+0000 7f3ddb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:23:23.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.607+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:23.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.607+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:23.613 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.607+0000 7f3ddb7fe640 10 monclient(hunting): my global_id is 4367 2024-03-25T14:23:23.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.607+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dcc003520 con 0x7f3de413f710 2024-03-25T14:23:23.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.614+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037298167 0 0) 0x7f3de40b28a0 con 0x7f3de41430c0 2024-03-25T14:23:23.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.614+0000 7f3ddb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:23:23.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.614+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:23.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.614+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:23.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.614+0000 7f3ddb7fe640 10 monclient(hunting): my global_id is 4387 2024-03-25T14:23:23.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.614+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de40b2a30 con 0x7f3de41430c0 2024-03-25T14:23:23.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.625+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374842611 0 0) 0x7f3de4142520 con 0x7f3de40a98f0 2024-03-25T14:23:23.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.625+0000 7f3ddb7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:23:23.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.625+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:23.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.625+0000 7f3ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:23.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.625+0000 7f3ddb7fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T14:23:23.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.625+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de40b28a0 con 0x7f3de40a98f0 2024-03-25T14:23:23.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567479929 0 0) 0x7f3dcc003520 con 0x7f3de413f710 2024-03-25T14:23:23.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 >> v1:172.21.15.153:6789/0 conn(0x7f3de41430c0 legacy=0x7f3de414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:23.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 >> v1:172.21.15.63:6789/0 conn(0x7f3de40a98f0 legacy=0x7f3de40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:23.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:23:23.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 1 monclient: found mon.noname-b 2024-03-25T14:23:23.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:23.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de40b44c0 con 0x7f3de413f710 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 10 monclient: _finish_auth 0 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3dd4001690 con 0x7f3de413f710 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3ddb7fe640 10 monclient: dump: 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:23.633 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:23.634 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:23.634 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:23.634 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:23.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3deb08f640 5 monclient: authenticate success, global_id 4367 2024-03-25T14:23:23.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.628+0000 7f3deb08f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:23.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dd40017f0 con 0x7f3de413f710 2024-03-25T14:23:23.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3ddb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:23.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3ddb7fe640 1 -- 172.21.15.63:0/4102515317 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3dd4001c00 con 0x7f3de413f710 2024-03-25T14:23:23.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3deb08f640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:23.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3deb08f640 4 set_mon_vals no callback set 2024-03-25T14:23:23.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3ddb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3ddb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3ddb7fe640 10 monclient: dump: 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:23.641 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:23.642 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:23.642 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:23.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.637+0000 7f3deb08f640 10 monclient: shutdown 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.638+0000 7f3deb08f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.638+0000 7f3deb08f640 1 -- 172.21.15.63:0/4102515317 >> v1:172.21.15.63:6790/0 conn(0x7f3de413f710 legacy=0x7f3de41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.638+0000 7f3deb08f640 1 -- 172.21.15.63:0/4102515317 shutdown_connections 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.638+0000 7f3deb08f640 1 -- 172.21.15.63:0/4102515317 >> 172.21.15.63:0/4102515317 conn(0x7f3de40b7ea0 msgr2=0x7f3de40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.638+0000 7f3deb08f640 1 -- 172.21.15.63:0/4102515317 shutdown_connections 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.638+0000 7f3deb08f640 1 -- 172.21.15.63:0/4102515317 wait complete. 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.638+0000 7f3deb08f640 10 monclient: build_initial_monmap 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 10 monclient: monmap: 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:23.644 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 1 librados: starting msgr at 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 1 librados: starting objecter 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 1 Processor -- start 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 1 -- start start 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 1 librados: setting wanted keys 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 1 librados: calling monclient init 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.639+0000 7f3deb08f640 10 monclient: init 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:23.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 10 monclient(hunting): picked mon.noname-a con 0x7f3de40a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 10 monclient(hunting): picked mon.noname-b con 0x7f3de413f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 10 monclient(hunting): picked mon.noname-c con 0x7f3de41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40ab8b0 con 0x7f3de40a98f0 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40abb90 con 0x7f3de413f710 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de4137870 con 0x7f3de41430c0 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:23.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.640+0000 7f3deb08f640 10 monclient(hunting): authenticate will time out at 1566.784848s 2024-03-25T14:23:23.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.650+0000 7f3ddbfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3de413f710 0x7f3de41df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36748/0 (socket says 172.21.15.63:36748) 2024-03-25T14:23:23.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.650+0000 7f3ddbfff640 1 -- 172.21.15.63:0/812203976 learned_addr learned my addr 172.21.15.63:0/812203976 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:23.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.653+0000 7f3dd97fa640 10 client.?.objecter ms_handle_connect 0x7f3de413f710 2024-03-25T14:23:23.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.653+0000 7f3dd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:23.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.658+0000 7f3dd97fa640 10 client.?.objecter ms_handle_connect 0x7f3de40a98f0 2024-03-25T14:23:23.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.658+0000 7f3dd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:23.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.663+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49922629 0 0) 0x7f3de40ab8b0 con 0x7f3de40a98f0 2024-03-25T14:23:23.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.663+0000 7f3dd97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:23:23.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.663+0000 7f3dd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:23.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.663+0000 7f3dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:23.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.663+0000 7f3dd97fa640 10 monclient(hunting): my global_id is 4380 2024-03-25T14:23:23.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.663+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd0004eb0 con 0x7f3de40a98f0 2024-03-25T14:23:23.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338637588 0 0) 0x7f3de40abb90 con 0x7f3de413f710 2024-03-25T14:23:23.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:23:23.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:23.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:23.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 10 monclient(hunting): my global_id is 4390 2024-03-25T14:23:23.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de40ab8b0 con 0x7f3de413f710 2024-03-25T14:23:23.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 10 client.?.objecter ms_handle_connect 0x7f3de41430c0 2024-03-25T14:23:23.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.667+0000 7f3dd97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:23.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897931934 0 0) 0x7f3de40ab8b0 con 0x7f3de413f710 2024-03-25T14:23:23.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3de40abb90 con 0x7f3de413f710 2024-03-25T14:23:23.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3dd4001b80 con 0x7f3de413f710 2024-03-25T14:23:23.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): dump: 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:23.690 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443505394 0 0) 0x7f3de4137870 con 0x7f3de41430c0 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T14:23:23.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.685+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de40ab8b0 con 0x7f3de41430c0 2024-03-25T14:23:23.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.692+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3509410295 0 0) 0x7f3dd0004eb0 con 0x7f3de40a98f0 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.692+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3de4137870 con 0x7f3de40a98f0 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.692+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3de00018f0 con 0x7f3de40a98f0 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.692+0000 7f3dd97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.692+0000 7f3dd97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.692+0000 7f3dd97fa640 10 monclient(hunting): dump: 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:23.696 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:23.697 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:23.697 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:23.697 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:23.697 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:23.697 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3744361069 0 0) 0x7f3de40abb90 con 0x7f3de413f710 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 >> v1:172.21.15.63:6790/0 conn(0x7f3de41430c0 legacy=0x7f3de41e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 >> v1:172.21.15.63:6789/0 conn(0x7f3de40a98f0 legacy=0x7f3de41350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 10 monclient: _finish_hunting 0 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 1 monclient: found mon.b 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de41389e0 con 0x7f3de413f710 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 10 monclient: _finish_auth 0 2024-03-25T14:23:23.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 10 monclient: _check_auth_tickets 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3dd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 5 monclient: authenticate success, global_id 4390 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 10 monclient: _renew_subs 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de4136d90 con 0x7f3de413f710 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 10 monclient: _renew_subs 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de41384c0 con 0x7f3de413f710 2024-03-25T14:23:23.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.705+0000 7f3deb08f640 1 librados: init done 2024-03-25T14:23:23.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.706+0000 7f3deb08f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:23.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.706+0000 7f3deb08f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:23.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.706+0000 7f3deb08f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:23.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:23.706+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de4136f80 con 0x7f3de413f710 2024-03-25T14:23:24.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dd40011f0 con 0x7f3de413f710 2024-03-25T14:23:24.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3dd97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:24.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f3dd4003b00 con 0x7f3de413f710 2024-03-25T14:23:24.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3dd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:24.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3dd97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3dd97fa640 10 monclient: dump: 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:24.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3ddaffd640 4 set_mon_vals no callback set 2024-03-25T14:23:24.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.020+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f3dd4003d20 con 0x7f3de413f710 2024-03-25T14:23:24.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.021+0000 7f3de8e2b640 10 monclient: get_auth_request con 0x7f3dd006df70 auth_method 0 2024-03-25T14:23:24.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.021+0000 7f3dd97fa640 10 client.4390.objecter ms_handle_connect 0x7f3dd006df70 2024-03-25T14:23:24.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.025+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f3dd40548f0 con 0x7f3de413f710 2024-03-25T14:23:24.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.025+0000 7f3dd97fa640 10 client.4390.objecter ms_dispatch 0x7f3de4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:24.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.025+0000 7f3dd97fa640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:24.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.025+0000 7f3dd97fa640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:24.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.025+0000 7f3dd97fa640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T14:23:24.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.026+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3dd4050400 con 0x7f3de413f710 2024-03-25T14:23:24.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.026+0000 7f3dd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:24.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.026+0000 7f3dd97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:24.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.279+0000 7f3deb08f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:24.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.280+0000 7f3deb08f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:24.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.280+0000 7f3deb08f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:24.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.280+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3de4138b70 con 0x7f3de413f710 2024-03-25T14:23:24.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.369+0000 7f3dd97fa640 1 -- 172.21.15.63:0/812203976 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f3dd4052330 con 0x7f3de413f710 2024-03-25T14:23:24.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.369+0000 7f3dd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:24.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.369+0000 7f3dd97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:24.373 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:24.373 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.371+0000 7f3deb08f640 10 librados: watch_flush enter 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 10 librados: watch_flush exit 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 10 client.4390.objecter successfully canceled tick 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 >> v1:172.21.15.153:6800/1688255552 conn(0x7f3dd006df70 legacy=0x7f3dd0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 10 monclient: shutdown 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 >> v1:172.21.15.153:6789/0 conn(0x7f3de413f710 legacy=0x7f3de41df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 shutdown_connections 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 >> 172.21.15.63:0/812203976 conn(0x7f3de40b7ea0 msgr2=0x7f3de4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 shutdown_connections 2024-03-25T14:23:24.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 1 -- 172.21.15.63:0/812203976 wait complete. 2024-03-25T14:23:24.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:24.372+0000 7f3deb08f640 1 librados: shutdown 2024-03-25T14:23:24.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:26.415 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:04.928 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.421 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:04.935 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.461 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:04.974 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.469 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:04.980 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.470 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:04.929 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.475 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:04.936 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.480 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:04.993 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.508 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:04.968 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:26.509 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:05.022 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:23:31.192 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:31.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.305+0000 7f2136740640 10 monclient: get_monmap_and_config 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.305+0000 7f2136740640 10 monclient: build_initial_monmap 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.305+0000 7f2136740640 10 monclient: monmap: 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:31.306162+0000 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:31.306162+0000 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:31.309 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.305+0000 7f2136740640 1 Processor -- start 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.305+0000 7f2136740640 1 -- start start 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.305+0000 7f2136740640 10 monclient: init 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 10 monclient(hunting): picked mon.noname-a con 0x7f21300a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 10 monclient(hunting): picked mon.noname-c con 0x7f213013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 10 monclient(hunting): picked mon.noname-b con 0x7f21301430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2130142520 con 0x7f21300a98f0 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21300b28a0 con 0x7f213013f710 2024-03-25T14:23:31.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21300b2a30 con 0x7f21301430c0 2024-03-25T14:23:31.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:31.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.306+0000 7f2136740640 10 monclient(hunting): authenticate will time out at 1574.451233s 2024-03-25T14:23:31.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.313+0000 7f212f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f213013f710 0x7f21301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:42494/0 (socket says 172.21.15.63:42494) 2024-03-25T14:23:31.316 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.313+0000 7f212f7fe640 1 -- 172.21.15.63:0/3270138344 learned_addr learned my addr 172.21.15.63:0/3270138344 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:31.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.350+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543623340 0 0) 0x7f2130142520 con 0x7f21300a98f0 2024-03-25T14:23:31.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.350+0000 7f212effd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:31.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.350+0000 7f212effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:31.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.350+0000 7f212effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:31.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.350+0000 7f212effd640 10 monclient(hunting): my global_id is 4383 2024-03-25T14:23:31.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.350+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2118003520 con 0x7f21300a98f0 2024-03-25T14:23:31.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1493651674 0 0) 0x7f2118003520 con 0x7f21300a98f0 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 >> v1:172.21.15.63:6790/0 conn(0x7f21301430c0 legacy=0x7f213014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 >> v1:172.21.15.153:6789/0 conn(0x7f213013f710 legacy=0x7f21301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 10 monclient: _finish_hunting 0 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 1 monclient: found mon.noname-a 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21300b44c0 con 0x7f21300a98f0 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 10 monclient: _finish_auth 0 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 10 monclient: _check_auth_tickets 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:31.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2120001690 con 0x7f21300a98f0 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f212effd640 10 monclient: dump: 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:31.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f2136740640 5 monclient: authenticate success, global_id 4383 2024-03-25T14:23:31.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.423+0000 7f2136740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21200017f0 con 0x7f21300a98f0 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f212effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f212effd640 1 -- 172.21.15.63:0/3270138344 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2120001c00 con 0x7f21300a98f0 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f2136740640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f2136740640 4 set_mon_vals no callback set 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f212effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f212effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.442+0000 7f212effd640 10 monclient: dump: 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:31.446 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:31.447 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:31.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.443+0000 7f2136740640 10 monclient: shutdown 2024-03-25T14:23:31.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.443+0000 7f2136740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:31.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.443+0000 7f2136740640 1 -- 172.21.15.63:0/3270138344 >> v1:172.21.15.63:6789/0 conn(0x7f21300a98f0 legacy=0x7f21300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:31.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.443+0000 7f2136740640 1 -- 172.21.15.63:0/3270138344 shutdown_connections 2024-03-25T14:23:31.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.443+0000 7f2136740640 1 -- 172.21.15.63:0/3270138344 >> 172.21.15.63:0/3270138344 conn(0x7f21300b7ea0 msgr2=0x7f21300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:31.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.443+0000 7f2136740640 1 -- 172.21.15.63:0/3270138344 shutdown_connections 2024-03-25T14:23:31.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.443+0000 7f2136740640 1 -- 172.21.15.63:0/3270138344 wait complete. 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 10 monclient: build_initial_monmap 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 10 monclient: monmap: 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:31.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 1 librados: starting msgr at 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 1 librados: starting objecter 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 1 Processor -- start 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 1 -- start start 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 1 librados: setting wanted keys 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 1 librados: calling monclient init 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.444+0000 7f2136740640 10 monclient: init 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 10 monclient(hunting): picked mon.noname-c con 0x7f21300a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 10 monclient(hunting): picked mon.noname-a con 0x7f213013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 10 monclient(hunting): picked mon.noname-b con 0x7f21301430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21300abd30 con 0x7f213013f710 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21301d77a0 con 0x7f21301430c0 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21301d51f0 con 0x7f21300a98f0 2024-03-25T14:23:31.451 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:31.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.445+0000 7f2136740640 10 monclient(hunting): authenticate will time out at 1574.590373s 2024-03-25T14:23:31.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.453+0000 7f212f7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f213013f710 0x7f2130138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:40380/0 (socket says 172.21.15.63:40380) 2024-03-25T14:23:31.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.453+0000 7f212f7fe640 1 -- 172.21.15.63:0/3664407191 learned_addr learned my addr 172.21.15.63:0/3664407191 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:31.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.456+0000 7f212cff9640 10 client.?.objecter ms_handle_connect 0x7f213013f710 2024-03-25T14:23:31.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.456+0000 7f212cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:31.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.468+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178184907 0 0) 0x7f21300abd30 con 0x7f213013f710 2024-03-25T14:23:31.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.468+0000 7f212cff9640 10 monclient(hunting): _negotiate 2024-03-25T14:23:31.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.468+0000 7f212cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:31.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.468+0000 7f212cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:31.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.468+0000 7f212cff9640 10 monclient(hunting): my global_id is 4386 2024-03-25T14:23:31.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.468+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f211c004eb0 con 0x7f213013f710 2024-03-25T14:23:31.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2762830789 0 0) 0x7f211c004eb0 con 0x7f213013f710 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21300abd30 con 0x7f213013f710 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f21240018b0 con 0x7f213013f710 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 10 monclient(hunting): dump: 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:31.496 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:31.497 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:31.497 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:31.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:31.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 10 client.?.objecter ms_handle_connect 0x7f21301430c0 2024-03-25T14:23:31.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.492+0000 7f212cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:31.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.494+0000 7f212cff9640 10 client.?.objecter ms_handle_connect 0x7f21300a98f0 2024-03-25T14:23:31.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.494+0000 7f212cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:31.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3476017180 0 0) 0x7f21300abd30 con 0x7f213013f710 2024-03-25T14:23:31.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 >> v1:172.21.15.63:6790/0 conn(0x7f21300a98f0 legacy=0x7f21301d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:31.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 >> v1:172.21.15.153:6789/0 conn(0x7f21301430c0 legacy=0x7f21301e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:31.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 10 monclient: _finish_hunting 0 2024-03-25T14:23:31.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 1 monclient: found mon.a 2024-03-25T14:23:31.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:31.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21301d83d0 con 0x7f213013f710 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 10 monclient: _finish_auth 0 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 10 monclient: _check_auth_tickets 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f212cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 5 monclient: authenticate success, global_id 4386 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 10 monclient: _renew_subs 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2130138a50 con 0x7f213013f710 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 10 monclient: _renew_subs 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21301e35c0 con 0x7f213013f710 2024-03-25T14:23:31.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.570+0000 7f2136740640 1 librados: init done 2024-03-25T14:23:31.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.571+0000 7f2136740640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:31.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.571+0000 7f2136740640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:31.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.571+0000 7f2136740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:31.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.571+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21301d7990 con 0x7f213013f710 2024-03-25T14:23:31.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.924+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2124001a10 con 0x7f213013f710 2024-03-25T14:23:31.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.924+0000 7f212cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:31.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.924+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2124004a80 con 0x7f213013f710 2024-03-25T14:23:31.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.924+0000 7f212cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:31.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.924+0000 7f212cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:31.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.924+0000 7f212cff9640 10 monclient: dump: 2024-03-25T14:23:31.929 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:31.929 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:31.929 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.930 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:31.930 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:31.930 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:31.930 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:31.930 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:31.931 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:31.931 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:31.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.924+0000 7f212e7fc640 4 set_mon_vals no callback set 2024-03-25T14:23:31.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.925+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f2124004ca0 con 0x7f213013f710 2024-03-25T14:23:31.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.925+0000 7f212ffff640 10 monclient: get_auth_request con 0x7f211c06dbf0 auth_method 0 2024-03-25T14:23:31.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.926+0000 7f212cff9640 10 client.4386.objecter ms_handle_connect 0x7f211c06dbf0 2024-03-25T14:23:31.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f2124054910 con 0x7f213013f710 2024-03-25T14:23:31.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 10 client.4386.objecter ms_dispatch 0x7f2130000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:31.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:31.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:31.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T14:23:31.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2124050420 con 0x7f213013f710 2024-03-25T14:23:31.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:31.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:31.930+0000 7f212cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:32.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.174+0000 7f2136740640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:32.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.174+0000 7f2136740640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:32.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.174+0000 7f2136740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:32.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.174+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21300627b0 con 0x7f213013f710 2024-03-25T14:23:32.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.219+0000 7f212cff9640 1 -- 172.21.15.63:0/3664407191 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f2124052350 con 0x7f213013f710 2024-03-25T14:23:32.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.219+0000 7f212cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:32.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.219+0000 7f212cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:32.223 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:32.223 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:32.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 10 librados: watch_flush enter 2024-03-25T14:23:32.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 10 librados: watch_flush exit 2024-03-25T14:23:32.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T14:23:32.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 10 client.4386.objecter successfully canceled tick 2024-03-25T14:23:32.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 >> v1:172.21.15.153:6800/1688255552 conn(0x7f211c06dbf0 legacy=0x7f211c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:32.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 10 monclient: shutdown 2024-03-25T14:23:32.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:32.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 >> v1:172.21.15.63:6789/0 conn(0x7f213013f710 legacy=0x7f2130138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:32.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 shutdown_connections 2024-03-25T14:23:32.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 >> 172.21.15.63:0/3664407191 conn(0x7f21300b7ea0 msgr2=0x7f2130074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:32.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 shutdown_connections 2024-03-25T14:23:32.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.221+0000 7f2136740640 1 -- 172.21.15.63:0/3664407191 wait complete. 2024-03-25T14:23:32.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:32.222+0000 7f2136740640 1 librados: shutdown 2024-03-25T14:23:32.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:39.044 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.152+0000 7fd6ace8d640 10 monclient: get_monmap_and_config 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.152+0000 7fd6ace8d640 10 monclient: build_initial_monmap 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.152+0000 7fd6ace8d640 10 monclient: monmap: 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:39.153674+0000 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:39.153674+0000 2024-03-25T14:23:39.156 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:39.157 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:39.157 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:39.157 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:39.157 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:39.157 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:39.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.153+0000 7fd6ace8d640 1 Processor -- start 2024-03-25T14:23:39.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.153+0000 7fd6ace8d640 1 -- start start 2024-03-25T14:23:39.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.153+0000 7fd6ace8d640 10 monclient: init 2024-03-25T14:23:39.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a80705b0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a81402d0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a8149ae0 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a80718c0 con 0x7fd6a81402d0 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a80714c0 con 0x7fd6a8149ae0 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a8072b50 con 0x7fd6a80705b0 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:39.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.154+0000 7fd6ace8d640 10 monclient(hunting): authenticate will time out at 1582.298873s 2024-03-25T14:23:39.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.164+0000 7fd6a5d9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd6a81402d0 0x7fd6a8140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:45080/0 (socket says 172.21.15.63:45080) 2024-03-25T14:23:39.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.164+0000 7fd6a5d9b640 1 -- 172.21.15.63:0/2404431094 learned_addr learned my addr 172.21.15.63:0/2404431094 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:39.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.175+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216136433 0 0) 0x7fd6a80718c0 con 0x7fd6a81402d0 2024-03-25T14:23:39.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.175+0000 7fd6a559a640 10 monclient(hunting): _negotiate 2024-03-25T14:23:39.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.175+0000 7fd6a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:39.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.175+0000 7fd6a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:39.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.175+0000 7fd6a559a640 10 monclient(hunting): my global_id is 4389 2024-03-25T14:23:39.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.175+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd690003520 con 0x7fd6a81402d0 2024-03-25T14:23:39.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.186+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442269329 0 0) 0x7fd6a8072b50 con 0x7fd6a80705b0 2024-03-25T14:23:39.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.186+0000 7fd6a559a640 10 monclient(hunting): _negotiate 2024-03-25T14:23:39.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.186+0000 7fd6a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:39.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.186+0000 7fd6a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:39.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.186+0000 7fd6a559a640 10 monclient(hunting): my global_id is 4373 2024-03-25T14:23:39.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.186+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a80718c0 con 0x7fd6a80705b0 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1327437981 0 0) 0x7fd690003520 con 0x7fd6a81402d0 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6790/0 conn(0x7fd6a80705b0 legacy=0x7fd6a813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.153:6789/0 conn(0x7fd6a8149ae0 legacy=0x7fd6a814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 10 monclient: _finish_hunting 0 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 1 monclient: found mon.noname-a 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a813dd10 con 0x7fd6a81402d0 2024-03-25T14:23:39.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 10 monclient: _finish_auth 0 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 10 monclient: _check_auth_tickets 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd69c001690 con 0x7fd6a81402d0 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6a559a640 10 monclient: dump: 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:39.196 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:39.197 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:39.197 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:39.197 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:39.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6ace8d640 5 monclient: authenticate success, global_id 4389 2024-03-25T14:23:39.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.191+0000 7fd6ace8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:39.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a5d9b640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6789/0 conn(0x7fd6a81402d0 legacy=0x7fd6a8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd69c0017f0 con 0x7fd6a81402d0 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a5d9b640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6789/0 conn(0x7fd6a81402d0 legacy=0x7fd6a8140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a5d9b640 1 --1- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6789/0 conn(0x7fd6a81402d0 0x7fd6a8140730 :-1 s=OPENED pgs=24553 cs=1 l=1).handle_message read tag failed 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a5d9b640 1 --1- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6789/0 conn(0x7fd6a81402d0 0x7fd6a8140730 :-1 s=OPENED pgs=24553 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6ace8d640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6ace8d640 4 set_mon_vals no callback set 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a559a640 10 monclient: ms_handle_reset current mon v1:172.21.15.63:6789/0 2024-03-25T14:23:39.213 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a559a640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:39.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.209+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6789/0 conn(0x7fd6a81402d0 legacy=0x7fd6a8140730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:23:39.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 10 monclient: shutdown 2024-03-25T14:23:39.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:23:39.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 10 monclient(hunting): picked mon.b con 0x7fd6a8149ae0 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:39.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 10 monclient(hunting): picked mon.c con 0x7fd6a80705b0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:39.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 10 monclient(hunting): picked mon.a con 0x7fd69000b090 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:39.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd690003520 con 0x7fd69000b090 2024-03-25T14:23:39.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd6a8072b50 con 0x7fd6a8149ae0 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 1 -- 172.21.15.63:0/2404431094 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd690010eb0 con 0x7fd6a80705b0 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6a559a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6789/0 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 20 monclient(hunting): shutdown discarding 1 pending message(s) 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6790/0 conn(0x7fd6a80705b0 legacy=0x7fd690009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.153:6789/0 conn(0x7fd6a8149ae0 legacy=0x7fd690003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2404431094 >> v1:172.21.15.63:6789/0 conn(0x7fd69000b090 legacy=0x7fd69000d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2404431094 shutdown_connections 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2404431094 >> 172.21.15.63:0/2404431094 conn(0x7fd6a80b5d10 msgr2=0x7fd6a80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2404431094 shutdown_connections 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.210+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2404431094 wait complete. 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.211+0000 7fd6ace8d640 10 monclient: build_initial_monmap 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.211+0000 7fd6ace8d640 10 monclient: monmap: 2024-03-25T14:23:39.217 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 librados: starting msgr at 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 librados: starting objecter 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 Processor -- start 2024-03-25T14:23:39.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 -- start start 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 librados: setting wanted keys 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 librados: calling monclient init 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient: init 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient(hunting): picked mon.noname-b con 0x7fd69000b090 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a80705b0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a8149ae0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd690003520 con 0x7fd6a80705b0 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a8072b50 con 0x7fd69000b090 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd690010eb0 con 0x7fd6a8149ae0 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:39.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.212+0000 7fd6ace8d640 10 monclient(hunting): authenticate will time out at 1582.357426s 2024-03-25T14:23:39.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.216+0000 7fd6a659c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd69000b090 0x7fd6a81440f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:47238/0 (socket says 172.21.15.63:47238) 2024-03-25T14:23:39.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.216+0000 7fd6a659c640 1 -- 172.21.15.63:0/2075361656 learned_addr learned my addr 172.21.15.63:0/2075361656 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:39.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.217+0000 7fd696ffd640 10 client.?.objecter ms_handle_connect 0x7fd69000b090 2024-03-25T14:23:39.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.217+0000 7fd696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:39.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.221+0000 7fd696ffd640 10 client.?.objecter ms_handle_connect 0x7fd6a80705b0 2024-03-25T14:23:39.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.221+0000 7fd696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:39.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.224+0000 7fd696ffd640 10 client.?.objecter ms_handle_connect 0x7fd6a8149ae0 2024-03-25T14:23:39.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.224+0000 7fd696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:39.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.247+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190187774 0 0) 0x7fd690003520 con 0x7fd6a80705b0 2024-03-25T14:23:39.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.247+0000 7fd696ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:39.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.247+0000 7fd696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:39.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.247+0000 7fd696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:39.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.247+0000 7fd696ffd640 10 monclient(hunting): my global_id is 4392 2024-03-25T14:23:39.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.247+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd688004eb0 con 0x7fd6a80705b0 2024-03-25T14:23:39.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.313+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 982492 0 0) 0x7fd688004eb0 con 0x7fd6a80705b0 2024-03-25T14:23:39.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.313+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd690003520 con 0x7fd6a80705b0 2024-03-25T14:23:39.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.313+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd69c001690 con 0x7fd6a80705b0 2024-03-25T14:23:39.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.313+0000 7fd696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:39.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.313+0000 7fd696ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:23:39.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.313+0000 7fd696ffd640 10 monclient(hunting): dump: 2024-03-25T14:23:39.317 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:39.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.313+0000 7fd696ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3339228912 0 0) 0x7fd690003520 con 0x7fd6a80705b0 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 >> v1:172.21.15.63:6790/0 conn(0x7fd6a8149ae0 legacy=0x7fd6a81d53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 >> v1:172.21.15.153:6789/0 conn(0x7fd69000b090 legacy=0x7fd6a81440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 1 monclient: found mon.a 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6900121d0 con 0x7fd6a80705b0 2024-03-25T14:23:39.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 10 monclient: _finish_auth 0 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 5 monclient: authenticate success, global_id 4392 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 10 monclient: _renew_subs 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a81416e0 con 0x7fd6a80705b0 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 10 monclient: _renew_subs 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a81d5cd0 con 0x7fd6a80705b0 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.334+0000 7fd6ace8d640 1 librados: init done 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.335+0000 7fd6ace8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.335+0000 7fd6ace8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.335+0000 7fd6ace8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:39.340 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.335+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a81428d0 con 0x7fd6a80705b0 2024-03-25T14:23:39.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.557+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd69c004a90 con 0x7fd6a80705b0 2024-03-25T14:23:39.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.557+0000 7fd696ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:39.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.557+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd69c004da0 con 0x7fd6a80705b0 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.557+0000 7fd696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.557+0000 7fd696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.557+0000 7fd696ffd640 10 monclient: dump: 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:39.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.557+0000 7fd6a4d99640 4 set_mon_vals no callback set 2024-03-25T14:23:39.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.652+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd69c01ad70 con 0x7fd6a80705b0 2024-03-25T14:23:39.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.653+0000 7fd6a659c640 10 monclient: get_auth_request con 0x7fd68806dc00 auth_method 0 2024-03-25T14:23:39.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.653+0000 7fd696ffd640 10 client.4392.objecter ms_handle_connect 0x7fd68806dc00 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fd69c054940 con 0x7fd6a80705b0 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 10 client.4392.objecter ms_dispatch 0x7fd6a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd69c050280 con 0x7fd6a80705b0 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:39.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.662+0000 7fd696ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:39.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.912+0000 7fd6ace8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:39.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.912+0000 7fd6ace8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:39.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.912+0000 7fd6ace8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:23:39.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.912+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6a80627b0 con 0x7fd6a80705b0 2024-03-25T14:23:39.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.949+0000 7fd696ffd640 1 -- 172.21.15.63:0/2075361656 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fd69c052380 con 0x7fd6a80705b0 2024-03-25T14:23:39.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.949+0000 7fd696ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:39.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.949+0000 7fd696ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:39.954 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:39.954 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 10 librados: watch_flush enter 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 10 librados: watch_flush exit 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 10 client.4392.objecter successfully canceled tick 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd68806dc00 legacy=0x7fd6880700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 10 monclient: shutdown 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 >> v1:172.21.15.63:6789/0 conn(0x7fd6a80705b0 legacy=0x7fd6a81420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 shutdown_connections 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 >> 172.21.15.63:0/2075361656 conn(0x7fd6a80b5d10 msgr2=0x7fd6a813d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 shutdown_connections 2024-03-25T14:23:39.956 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 1 -- 172.21.15.63:0/2075361656 wait complete. 2024-03-25T14:23:39.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:39.952+0000 7fd6ace8d640 1 librados: shutdown 2024-03-25T14:23:39.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:46.777 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:46.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.887+0000 7f9013a7b640 10 monclient: get_monmap_and_config 2024-03-25T14:23:46.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.887+0000 7f9013a7b640 10 monclient: build_initial_monmap 2024-03-25T14:23:46.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.888+0000 7f9013a7b640 10 monclient: monmap: 2024-03-25T14:23:46.891 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:46.889083+0000 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:46.889083+0000 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:46.892 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:46.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.888+0000 7f9013a7b640 1 Processor -- start 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.888+0000 7f9013a7b640 1 -- start start 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.888+0000 7f9013a7b640 10 monclient: init 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f900c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f900c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f900c1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c142520 con 0x7f900c13f710 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c0b28a0 con 0x7f900c1430c0 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c0b2a30 con 0x7f900c0a98f0 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:46.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.889+0000 7f9013a7b640 10 monclient(hunting): authenticate will time out at 1590.034188s 2024-03-25T14:23:46.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.895+0000 7f9012018640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f900c1430c0 0x7f900c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:59650/0 (socket says 172.21.15.63:59650) 2024-03-25T14:23:46.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.895+0000 7f9012018640 1 -- 172.21.15.63:0/1101445914 learned_addr learned my addr 172.21.15.63:0/1101445914 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:46.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.919+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483565237 0 0) 0x7f900c0b28a0 con 0x7f900c1430c0 2024-03-25T14:23:46.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.919+0000 7f9010815640 10 monclient(hunting): _negotiate 2024-03-25T14:23:46.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.919+0000 7f9010815640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:46.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.919+0000 7f9010815640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:46.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.919+0000 7f9010815640 10 monclient(hunting): my global_id is 4399 2024-03-25T14:23:46.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.919+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8003520 con 0x7f900c1430c0 2024-03-25T14:23:46.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.929+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549966564 0 0) 0x7f900c142520 con 0x7f900c13f710 2024-03-25T14:23:46.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.929+0000 7f9010815640 10 monclient(hunting): _negotiate 2024-03-25T14:23:46.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.929+0000 7f9010815640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:46.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.929+0000 7f9010815640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:46.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.929+0000 7f9010815640 10 monclient(hunting): my global_id is 4395 2024-03-25T14:23:46.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.929+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c0b28a0 con 0x7f900c13f710 2024-03-25T14:23:46.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.941+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939307069 0 0) 0x7f900c0b2a30 con 0x7f900c0a98f0 2024-03-25T14:23:46.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.942+0000 7f9010815640 10 monclient(hunting): _negotiate 2024-03-25T14:23:46.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.942+0000 7f9010815640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:46.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.942+0000 7f9010815640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:46.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.942+0000 7f9010815640 10 monclient(hunting): my global_id is 4379 2024-03-25T14:23:46.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.942+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c142520 con 0x7f900c0a98f0 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946716699 0 0) 0x7f900c0b28a0 con 0x7f900c13f710 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 >> v1:172.21.15.63:6790/0 conn(0x7f900c0a98f0 legacy=0x7f900c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 >> v1:172.21.15.153:6789/0 conn(0x7f900c1430c0 legacy=0x7f900c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 10 monclient: _finish_hunting 0 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 1 monclient: found mon.noname-a 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:23:46.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c0b44c0 con 0x7f900c13f710 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 10 monclient: _finish_auth 0 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 10 monclient: _check_auth_tickets 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8ffc001670 con 0x7f900c13f710 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9010815640 10 monclient: dump: 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:46.959 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:46.960 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:46.960 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:46.960 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:46.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9013a7b640 5 monclient: authenticate success, global_id 4395 2024-03-25T14:23:46.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.954+0000 7f9013a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:46.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.982+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ffc0017d0 con 0x7f900c13f710 2024-03-25T14:23:46.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.982+0000 7f9010815640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:46.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.982+0000 7f9010815640 1 -- 172.21.15.63:0/1101445914 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8ffc001be0 con 0x7f900c13f710 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.982+0000 7f9010815640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.982+0000 7f9010815640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.982+0000 7f9010815640 10 monclient: dump: 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:46.987 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:46.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.983+0000 7f9013a7b640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:46.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.983+0000 7f9013a7b640 4 set_mon_vals no callback set 2024-03-25T14:23:46.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.983+0000 7f9013a7b640 10 monclient: shutdown 2024-03-25T14:23:46.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.983+0000 7f9013a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:46.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.983+0000 7f9013a7b640 1 -- 172.21.15.63:0/1101445914 >> v1:172.21.15.63:6789/0 conn(0x7f900c13f710 legacy=0x7f900c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:46.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.984+0000 7f9013a7b640 1 -- 172.21.15.63:0/1101445914 shutdown_connections 2024-03-25T14:23:46.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.984+0000 7f9013a7b640 1 -- 172.21.15.63:0/1101445914 >> 172.21.15.63:0/1101445914 conn(0x7f900c0b7ea0 msgr2=0x7f900c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:46.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.984+0000 7f9013a7b640 1 -- 172.21.15.63:0/1101445914 shutdown_connections 2024-03-25T14:23:46.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.984+0000 7f9013a7b640 1 -- 172.21.15.63:0/1101445914 wait complete. 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 10 monclient: build_initial_monmap 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 10 monclient: monmap: 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 1 librados: starting msgr at 2024-03-25T14:23:46.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 1 librados: starting objecter 2024-03-25T14:23:46.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 1 Processor -- start 2024-03-25T14:23:46.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 1 -- start start 2024-03-25T14:23:46.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 1 librados: setting wanted keys 2024-03-25T14:23:46.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 1 librados: calling monclient init 2024-03-25T14:23:46.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.985+0000 7f9013a7b640 10 monclient: init 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f900c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f900c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f900c1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c139dc0 con 0x7f900c0a98f0 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c13aff0 con 0x7f900c1430c0 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c13b2d0 con 0x7f900c13f710 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:46.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:46.986+0000 7f9013a7b640 10 monclient(hunting): authenticate will time out at 1590.131467s 2024-03-25T14:23:47.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.002+0000 7f9011016640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f900c13f710 0x7f900c1df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:34134/0 (socket says 172.21.15.63:34134) 2024-03-25T14:23:47.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.002+0000 7f9011016640 1 -- 172.21.15.63:0/1944554339 learned_addr learned my addr 172.21.15.63:0/1944554339 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:47.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.003+0000 7f90027fc640 10 client.?.objecter ms_handle_connect 0x7f900c13f710 2024-03-25T14:23:47.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.003+0000 7f90027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:47.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.007+0000 7f90027fc640 10 client.?.objecter ms_handle_connect 0x7f900c1430c0 2024-03-25T14:23:47.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.007+0000 7f90027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:47.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.012+0000 7f90027fc640 10 client.?.objecter ms_handle_connect 0x7f900c0a98f0 2024-03-25T14:23:47.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.012+0000 7f90027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:47.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.014+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063750960 0 0) 0x7f900c13aff0 con 0x7f900c1430c0 2024-03-25T14:23:47.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.014+0000 7f90027fc640 10 monclient(hunting): _negotiate 2024-03-25T14:23:47.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.014+0000 7f90027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:47.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.014+0000 7f90027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:47.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.014+0000 7f90027fc640 10 monclient(hunting): my global_id is 4402 2024-03-25T14:23:47.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.014+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0004eb0 con 0x7f900c1430c0 2024-03-25T14:23:47.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.016+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41041625 0 0) 0x7f900c13b2d0 con 0x7f900c13f710 2024-03-25T14:23:47.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.016+0000 7f90027fc640 10 monclient(hunting): _negotiate 2024-03-25T14:23:47.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.016+0000 7f90027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:47.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.016+0000 7f90027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:47.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.016+0000 7f90027fc640 10 monclient(hunting): my global_id is 4382 2024-03-25T14:23:47.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.016+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c13aff0 con 0x7f900c13f710 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.027+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 307898392 0 0) 0x7f8ff0004eb0 con 0x7f900c1430c0 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.027+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f900c13b2d0 con 0x7f900c1430c0 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.027+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9004001af0 con 0x7f900c1430c0 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.027+0000 7f90027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.027+0000 7f90027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.027+0000 7f90027fc640 10 monclient(hunting): dump: 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:47.031 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:47.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.027+0000 7f90027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:47.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.030+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500904511 0 0) 0x7f900c139dc0 con 0x7f900c0a98f0 2024-03-25T14:23:47.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.030+0000 7f90027fc640 10 monclient(hunting): _negotiate 2024-03-25T14:23:47.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.030+0000 7f90027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:47.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.030+0000 7f90027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:47.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.030+0000 7f90027fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T14:23:47.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.030+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0004eb0 con 0x7f900c0a98f0 2024-03-25T14:23:47.043 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.040+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1695805106 0 0) 0x7f900c13aff0 con 0x7f900c13f710 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.040+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f900c139dc0 con 0x7f900c13f710 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.040+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8ffc001670 con 0x7f900c13f710 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.040+0000 7f90027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.040+0000 7f90027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.040+0000 7f90027fc640 10 monclient(hunting): dump: 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:47.044 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:47.045 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:47.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3305538291 0 0) 0x7f900c13b2d0 con 0x7f900c1430c0 2024-03-25T14:23:47.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 >> v1:172.21.15.63:6790/0 conn(0x7f900c13f710 legacy=0x7f900c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:47.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 >> v1:172.21.15.63:6789/0 conn(0x7f900c0a98f0 legacy=0x7f900c12f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:47.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 10 monclient: _finish_hunting 0 2024-03-25T14:23:47.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 1 monclient: found mon.b 2024-03-25T14:23:47.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c0b6ca0 con 0x7f900c1430c0 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 10 monclient: _finish_auth 0 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 10 monclient: _check_auth_tickets 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f90027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f9013a7b640 5 monclient: authenticate success, global_id 4402 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f9013a7b640 10 monclient: _renew_subs 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f9013a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.041+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c139810 con 0x7f900c1430c0 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.042+0000 7f9013a7b640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.042+0000 7f9013a7b640 10 monclient: _renew_subs 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.042+0000 7f9013a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.042+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c0b7940 con 0x7f900c1430c0 2024-03-25T14:23:47.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.042+0000 7f9013a7b640 1 librados: init done 2024-03-25T14:23:47.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.043+0000 7f9013a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:47.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.043+0000 7f9013a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:47.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.043+0000 7f9013a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:47.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.043+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900c0627b0 con 0x7f900c1430c0 2024-03-25T14:23:47.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.316+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9004001f20 con 0x7f900c1430c0 2024-03-25T14:23:47.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.316+0000 7f90027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:47.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.316+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f9004004d00 con 0x7f900c1430c0 2024-03-25T14:23:47.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.316+0000 7f90027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.316+0000 7f90027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.316+0000 7f90027fc640 10 monclient: dump: 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:47.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.316+0000 7f9003fff640 4 set_mon_vals no callback set 2024-03-25T14:23:47.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.317+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f900401acd0 con 0x7f900c1430c0 2024-03-25T14:23:47.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.318+0000 7f9011817640 10 monclient: get_auth_request con 0x7f8ff006df70 auth_method 0 2024-03-25T14:23:47.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.318+0000 7f90027fc640 10 client.4402.objecter ms_handle_connect 0x7f8ff006df70 2024-03-25T14:23:47.326 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.323+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f90040549e0 con 0x7f900c1430c0 2024-03-25T14:23:47.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.323+0000 7f90027fc640 10 client.4402.objecter ms_dispatch 0x7f900c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:47.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.323+0000 7f90027fc640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:47.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.323+0000 7f90027fc640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:47.327 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.323+0000 7f90027fc640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T14:23:47.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.325+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9004050320 con 0x7f900c1430c0 2024-03-25T14:23:47.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.325+0000 7f90027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:47.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.325+0000 7f90027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.573+0000 7f9013a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.574+0000 7f9013a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.574+0000 7f9013a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:23:47.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.574+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f900c074cc0 con 0x7f900c1430c0 2024-03-25T14:23:47.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.740+0000 7f90027fc640 1 -- 172.21.15.63:0/1944554339 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f9004052420 con 0x7f900c1430c0 2024-03-25T14:23:47.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.740+0000 7f90027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:47.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.740+0000 7f90027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:47.744 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:47.744 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:47.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 10 librados: watch_flush enter 2024-03-25T14:23:47.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 10 librados: watch_flush exit 2024-03-25T14:23:47.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T14:23:47.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 10 client.4402.objecter successfully canceled tick 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 >> v1:172.21.15.153:6800/1688255552 conn(0x7f8ff006df70 legacy=0x7f8ff0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 10 monclient: shutdown 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 >> v1:172.21.15.153:6789/0 conn(0x7f900c1430c0 legacy=0x7f900c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 shutdown_connections 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 >> 172.21.15.63:0/1944554339 conn(0x7f900c0b7ea0 msgr2=0x7f900c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 shutdown_connections 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 1 -- 172.21.15.63:0/1944554339 wait complete. 2024-03-25T14:23:47.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:47.742+0000 7f9013a7b640 1 librados: shutdown 2024-03-25T14:23:47.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:23:54.566 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:23:54.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.678+0000 7ff9cddb5640 10 monclient: get_monmap_and_config 2024-03-25T14:23:54.682 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.678+0000 7ff9cddb5640 10 monclient: build_initial_monmap 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.678+0000 7ff9cddb5640 10 monclient: monmap: 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:23:54.679964+0000 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:23:54.679964+0000 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:23:54.683 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:54.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.679+0000 7ff9cddb5640 1 Processor -- start 2024-03-25T14:23:54.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.680+0000 7ff9cddb5640 1 -- start start 2024-03-25T14:23:54.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.680+0000 7ff9cddb5640 10 monclient: init 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.680+0000 7ff9cddb5640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.680+0000 7ff9cddb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.680+0000 7ff9cddb5640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c80b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.681+0000 7ff9cddb5640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c80b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.681+0000 7ff9cddb5640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.681+0000 7ff9cddb5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80af940 con 0x7ff9c80b1730 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.681+0000 7ff9cddb5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80b0a70 con 0x7ff9c80b4d60 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.681+0000 7ff9cddb5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80b0d50 con 0x7ff9c814be90 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.681+0000 7ff9cddb5640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:54.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.681+0000 7ff9cddb5640 10 monclient(hunting): authenticate will time out at 1597.825642s 2024-03-25T14:23:54.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.695+0000 7ff9c77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff9c80b4d60 0x7ff9c813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:55012/0 (socket says 172.21.15.63:55012) 2024-03-25T14:23:54.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.695+0000 7ff9bffff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff9c80b1730 0x7ff9c80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:41356/0 (socket says 172.21.15.63:41356) 2024-03-25T14:23:54.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.695+0000 7ff9c77fe640 1 -- 172.21.15.63:0/3789144926 learned_addr learned my addr 172.21.15.63:0/3789144926 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:54.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.702+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994890652 0 0) 0x7ff9c80b0a70 con 0x7ff9c80b4d60 2024-03-25T14:23:54.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.702+0000 7ff9c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:54.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.702+0000 7ff9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:54.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.702+0000 7ff9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:54.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.702+0000 7ff9c6ffd640 10 monclient(hunting): my global_id is 4405 2024-03-25T14:23:54.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.702+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b4003520 con 0x7ff9c80b4d60 2024-03-25T14:23:54.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.709+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100485653 0 0) 0x7ff9c80af940 con 0x7ff9c80b1730 2024-03-25T14:23:54.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.709+0000 7ff9c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:54.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.709+0000 7ff9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:54.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.709+0000 7ff9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:54.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.709+0000 7ff9c6ffd640 10 monclient(hunting): my global_id is 4401 2024-03-25T14:23:54.713 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.709+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c80b0a70 con 0x7ff9c80b1730 2024-03-25T14:23:54.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.710+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652553188 0 0) 0x7ff9c80b0d50 con 0x7ff9c814be90 2024-03-25T14:23:54.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.710+0000 7ff9c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:23:54.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.710+0000 7ff9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:54.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.710+0000 7ff9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:54.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.710+0000 7ff9c6ffd640 10 monclient(hunting): my global_id is 4385 2024-03-25T14:23:54.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.710+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c80af940 con 0x7ff9c814be90 2024-03-25T14:23:54.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4082903961 0 0) 0x7ff9b4003520 con 0x7ff9c80b4d60 2024-03-25T14:23:54.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 >> v1:172.21.15.63:6790/0 conn(0x7ff9c814be90 legacy=0x7ff9c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:54.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 >> v1:172.21.15.63:6789/0 conn(0x7ff9c80b1730 legacy=0x7ff9c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:54.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:23:54.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 1 monclient: found mon.noname-c 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c80b1030 con 0x7ff9c80b4d60 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 10 monclient: _finish_auth 0 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff9b8001a10 con 0x7ff9c80b4d60 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9cddb5640 5 monclient: authenticate success, global_id 4405 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9cddb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9c6ffd640 10 monclient: dump: 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:54.724 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:54.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.719+0000 7ff9cddb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:23:54.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.730+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9b8001b70 con 0x7ff9c80b4d60 2024-03-25T14:23:54.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:54.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9c6ffd640 1 -- 172.21.15.63:0/3789144926 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff9b8001f80 con 0x7ff9c80b4d60 2024-03-25T14:23:54.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9cddb5640 10 monclient: get_monmap_and_config success 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9cddb5640 4 set_mon_vals no callback set 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9c6ffd640 10 monclient: dump: 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:54.735 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9cddb5640 10 monclient: shutdown 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9cddb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.731+0000 7ff9cddb5640 1 -- 172.21.15.63:0/3789144926 >> v1:172.21.15.153:6789/0 conn(0x7ff9c80b4d60 legacy=0x7ff9c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.732+0000 7ff9cddb5640 1 -- 172.21.15.63:0/3789144926 shutdown_connections 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.732+0000 7ff9cddb5640 1 -- 172.21.15.63:0/3789144926 >> 172.21.15.63:0/3789144926 conn(0x7ff9c80617a0 msgr2=0x7ff9c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.732+0000 7ff9cddb5640 1 -- 172.21.15.63:0/3789144926 shutdown_connections 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.732+0000 7ff9cddb5640 1 -- 172.21.15.63:0/3789144926 wait complete. 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.732+0000 7ff9cddb5640 10 monclient: build_initial_monmap 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.732+0000 7ff9cddb5640 10 monclient: monmap: 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:23:54.738 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 librados: starting msgr at 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 librados: starting objecter 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 Processor -- start 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 -- start start 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 librados: setting wanted keys 2024-03-25T14:23:54.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 librados: calling monclient init 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient: init 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient: _reopen_session rank -1 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient(hunting): picked mon.noname-b con 0x7ff9c80b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient(hunting): picked mon.noname-a con 0x7ff9c80b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c8135810 con 0x7ff9c80b4d60 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c813b330 con 0x7ff9c80b1730 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c81d5230 con 0x7ff9c814be90 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient(hunting): _renew_subs 2024-03-25T14:23:54.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.733+0000 7ff9cddb5640 10 monclient(hunting): authenticate will time out at 1597.878418s 2024-03-25T14:23:54.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.746+0000 7ff9c77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff9c80b1730 0x7ff9c81d7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:55034/0 (socket says 172.21.15.63:55034) 2024-03-25T14:23:54.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.746+0000 7ff9c77fe640 1 -- 172.21.15.63:0/734969389 learned_addr learned my addr 172.21.15.63:0/734969389 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:23:54.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.753+0000 7ff9c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff9c814be90 2024-03-25T14:23:54.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.753+0000 7ff9c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:54.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.754+0000 7ff9c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff9c80b4d60 2024-03-25T14:23:54.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.754+0000 7ff9c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:54.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.757+0000 7ff9c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff9c80b1730 2024-03-25T14:23:54.760 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.757+0000 7ff9c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:23:54.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.762+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862950197 0 0) 0x7ff9c813b330 con 0x7ff9c80b1730 2024-03-25T14:23:54.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.762+0000 7ff9c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:23:54.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.762+0000 7ff9c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:54.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.762+0000 7ff9c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:54.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.762+0000 7ff9c4ff9640 10 monclient(hunting): my global_id is 4408 2024-03-25T14:23:54.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.762+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a8004eb0 con 0x7ff9c80b1730 2024-03-25T14:23:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.769+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425234818 0 0) 0x7ff9c81d5230 con 0x7ff9c814be90 2024-03-25T14:23:54.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.769+0000 7ff9c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:23:54.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.769+0000 7ff9c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:54.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.769+0000 7ff9c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:54.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.769+0000 7ff9c4ff9640 10 monclient(hunting): my global_id is 4388 2024-03-25T14:23:54.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.769+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c813b330 con 0x7ff9c814be90 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050363802 0 0) 0x7ff9c8135810 con 0x7ff9c80b4d60 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c81d5230 con 0x7ff9c80b4d60 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 740437335 0 0) 0x7ff9c813b330 con 0x7ff9c814be90 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9c8135810 con 0x7ff9c814be90 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff9c00017d0 con 0x7ff9c814be90 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): dump: 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:54.784 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:54.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.780+0000 7ff9c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:23:54.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.785+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1059969236 0 0) 0x7ff9a8004eb0 con 0x7ff9c80b1730 2024-03-25T14:23:54.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.785+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9c813b330 con 0x7ff9c80b1730 2024-03-25T14:23:54.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.785+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff9b8001f40 con 0x7ff9c80b1730 2024-03-25T14:23:54.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.785+0000 7ff9c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:54.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.785+0000 7ff9c4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.785+0000 7ff9c4ff9640 10 monclient(hunting): dump: 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:54.790 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:54.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1022830021 0 0) 0x7ff9c8135810 con 0x7ff9c814be90 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 >> v1:172.21.15.153:6789/0 conn(0x7ff9c80b1730 legacy=0x7ff9c81d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 >> v1:172.21.15.63:6789/0 conn(0x7ff9c80b4d60 legacy=0x7ff9c81349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 1 monclient: found mon.c 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8139c30 con 0x7ff9c814be90 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 10 monclient: _finish_auth 0 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 5 monclient: authenticate success, global_id 4388 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 10 monclient: _renew_subs 2024-03-25T14:23:54.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8139dc0 con 0x7ff9c814be90 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 10 monclient: _renew_subs 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c8139270 con 0x7ff9c814be90 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.796+0000 7ff9cddb5640 1 librados: init done 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.797+0000 7ff9cddb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:54.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.797+0000 7ff9cddb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:54.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.797+0000 7ff9cddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:54.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:54.797+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c81401f0 con 0x7ff9c814be90 2024-03-25T14:23:55.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.013+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9c0001c60 con 0x7ff9c814be90 2024-03-25T14:23:55.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.013+0000 7ff9c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:23:55.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.013+0000 7ff9c67fc640 4 set_mon_vals no callback set 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.119+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff9c00037b0 con 0x7ff9c814be90 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.119+0000 7ff9c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.119+0000 7ff9c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.119+0000 7ff9c4ff9640 10 monclient: dump: 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:23:55.123 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:23:55.124 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:23:55.124 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:23:55.124 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:23:55.124 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:23:55.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.120+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7ff9c00039d0 con 0x7ff9c814be90 2024-03-25T14:23:55.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.120+0000 7ff9c77fe640 10 monclient: get_auth_request con 0x7ff9a806df20 auth_method 0 2024-03-25T14:23:55.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.120+0000 7ff9c4ff9640 10 client.4388.objecter ms_handle_connect 0x7ff9a806df20 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7ff9c0053460 con 0x7ff9c814be90 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 10 client.4388.objecter ms_dispatch 0x7ff9c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9c004ef70 con 0x7ff9c814be90 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:23:55.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.129+0000 7ff9c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:23:55.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.360+0000 7ff9cddb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:55.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.360+0000 7ff9cddb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:55.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.360+0000 7ff9cddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:23:55.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.360+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9c80627b0 con 0x7ff9c814be90 2024-03-25T14:23:55.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.425+0000 7ff9c4ff9640 1 -- 172.21.15.63:0/734969389 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7ff9c0050ea0 con 0x7ff9c814be90 2024-03-25T14:23:55.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.425+0000 7ff9c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:23:55.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.425+0000 7ff9c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:23:55.430 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:23:55.430 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:23:55.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 10 librados: watch_flush enter 2024-03-25T14:23:55.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 10 librados: watch_flush exit 2024-03-25T14:23:55.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T14:23:55.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 10 client.4388.objecter successfully canceled tick 2024-03-25T14:23:55.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 >> v1:172.21.15.153:6800/1688255552 conn(0x7ff9a806df20 legacy=0x7ff9a80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:55.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 10 monclient: shutdown 2024-03-25T14:23:55.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:23:55.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 >> v1:172.21.15.63:6790/0 conn(0x7ff9c814be90 legacy=0x7ff9c8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:23:55.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 shutdown_connections 2024-03-25T14:23:55.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 >> 172.21.15.63:0/734969389 conn(0x7ff9c80617a0 msgr2=0x7ff9c80b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:23:55.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 shutdown_connections 2024-03-25T14:23:55.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 1 -- 172.21.15.63:0/734969389 wait complete. 2024-03-25T14:23:55.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:23:55.428+0000 7ff9cddb5640 1 librados: shutdown 2024-03-25T14:23:55.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:02.251 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:02.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.368+0000 7f6191c81640 10 monclient: get_monmap_and_config 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.368+0000 7f6191c81640 10 monclient: build_initial_monmap 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.368+0000 7f6191c81640 10 monclient: monmap: 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:02.369237+0000 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:02.369237+0000 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:02.372 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:02.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.368+0000 7f6191c81640 1 Processor -- start 2024-03-25T14:24:02.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 1 -- start start 2024-03-25T14:24:02.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient: init 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient(hunting): picked mon.noname-a con 0x7f618c0b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient(hunting): picked mon.noname-b con 0x7f618c0b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient(hunting): picked mon.noname-c con 0x7f618c14be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c0af940 con 0x7f618c0b4d60 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c0b0a70 con 0x7f618c14be90 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c0b0d50 con 0x7f618c0b1730 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:02.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.369+0000 7f6191c81640 10 monclient(hunting): authenticate will time out at 1605.514287s 2024-03-25T14:24:02.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.379+0000 7f618affd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f618c0b1730 0x7f618c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:48602/0 (socket says 172.21.15.63:48602) 2024-03-25T14:24:02.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.379+0000 7f618affd640 1 -- 172.21.15.63:0/2887260503 learned_addr learned my addr 172.21.15.63:0/2887260503 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:02.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.400+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112344007 0 0) 0x7f618c0b0d50 con 0x7f618c0b1730 2024-03-25T14:24:02.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.400+0000 7f618a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:02.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.400+0000 7f618a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:02.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.400+0000 7f618a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:02.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.400+0000 7f618a7fc640 10 monclient(hunting): my global_id is 4391 2024-03-25T14:24:02.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.400+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174003520 con 0x7f618c0b1730 2024-03-25T14:24:02.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.402+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130375662 0 0) 0x7f618c0b0a70 con 0x7f618c14be90 2024-03-25T14:24:02.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.402+0000 7f618a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:02.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.402+0000 7f618a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:02.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.402+0000 7f618a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:02.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.402+0000 7f618a7fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T14:24:02.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.402+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618c0b0d50 con 0x7f618c14be90 2024-03-25T14:24:02.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.403+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174290672 0 0) 0x7f618c0af940 con 0x7f618c0b4d60 2024-03-25T14:24:02.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.403+0000 7f618a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:02.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.403+0000 7f618a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:02.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.403+0000 7f618a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:02.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.403+0000 7f618a7fc640 10 monclient(hunting): my global_id is 4407 2024-03-25T14:24:02.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.403+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618c0b0a70 con 0x7f618c0b4d60 2024-03-25T14:24:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1237305899 0 0) 0x7f6174003520 con 0x7f618c0b1730 2024-03-25T14:24:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 >> v1:172.21.15.153:6789/0 conn(0x7f618c14be90 legacy=0x7f618c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 >> v1:172.21.15.63:6789/0 conn(0x7f618c0b4d60 legacy=0x7f618c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:24:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 1 monclient: found mon.noname-b 2024-03-25T14:24:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:02.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c0b1030 con 0x7f618c0b1730 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6180001690 con 0x7f618c0b1730 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f6191c81640 5 monclient: authenticate success, global_id 4391 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f618a7fc640 10 monclient: dump: 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:02.420 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:02.421 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:02.421 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:02.421 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:02.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.415+0000 7f6191c81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61800017f0 con 0x7f618c0b1730 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f618a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f618a7fc640 1 -- 172.21.15.63:0/2887260503 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6180001c00 con 0x7f618c0b1730 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f6191c81640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f6191c81640 4 set_mon_vals no callback set 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f618a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f618a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.430+0000 7f618a7fc640 10 monclient: dump: 2024-03-25T14:24:02.434 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:02.435 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:02.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.431+0000 7f6191c81640 10 monclient: shutdown 2024-03-25T14:24:02.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.431+0000 7f6191c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:02.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.431+0000 7f6191c81640 1 -- 172.21.15.63:0/2887260503 >> v1:172.21.15.63:6790/0 conn(0x7f618c0b1730 legacy=0x7f618c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.431+0000 7f6191c81640 1 -- 172.21.15.63:0/2887260503 shutdown_connections 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.431+0000 7f6191c81640 1 -- 172.21.15.63:0/2887260503 >> 172.21.15.63:0/2887260503 conn(0x7f618c0617a0 msgr2=0x7f618c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.431+0000 7f6191c81640 1 -- 172.21.15.63:0/2887260503 shutdown_connections 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.431+0000 7f6191c81640 1 -- 172.21.15.63:0/2887260503 wait complete. 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 10 monclient: build_initial_monmap 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 10 monclient: monmap: 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:02.438 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 1 librados: starting msgr at 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 1 librados: starting objecter 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 1 Processor -- start 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 1 -- start start 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 1 librados: setting wanted keys 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 1 librados: calling monclient init 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.432+0000 7f6191c81640 10 monclient: init 2024-03-25T14:24:02.439 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 10 monclient(hunting): picked mon.noname-a con 0x7f618c0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 10 monclient(hunting): picked mon.noname-b con 0x7f618c0b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 10 monclient(hunting): picked mon.noname-c con 0x7f618c14be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c1406b0 con 0x7f618c0b1730 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c141a70 con 0x7f618c0b4d60 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c135810 con 0x7f618c14be90 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:02.440 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.433+0000 7f6191c81640 10 monclient(hunting): authenticate will time out at 1605.578128s 2024-03-25T14:24:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.436+0000 7f618b7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f618c0b1730 0x7f618c1daca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36388/0 (socket says 172.21.15.63:36388) 2024-03-25T14:24:02.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.436+0000 7f618b7fe640 1 -- 172.21.15.63:0/2355095433 learned_addr learned my addr 172.21.15.63:0/2355095433 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:02.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.440+0000 7f6173fff640 10 client.?.objecter ms_handle_connect 0x7f618c0b1730 2024-03-25T14:24:02.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.440+0000 7f6173fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:02.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.446+0000 7f6173fff640 10 client.?.objecter ms_handle_connect 0x7f618c0b4d60 2024-03-25T14:24:02.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.446+0000 7f6173fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:02.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.452+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304218869 0 0) 0x7f618c141a70 con 0x7f618c0b4d60 2024-03-25T14:24:02.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.452+0000 7f6173fff640 10 monclient(hunting): _negotiate 2024-03-25T14:24:02.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.452+0000 7f6173fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:02.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.452+0000 7f6173fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:02.455 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.452+0000 7f6173fff640 10 monclient(hunting): my global_id is 4414 2024-03-25T14:24:02.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.452+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6178004eb0 con 0x7f618c0b4d60 2024-03-25T14:24:02.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.453+0000 7f6173fff640 10 client.?.objecter ms_handle_connect 0x7f618c14be90 2024-03-25T14:24:02.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.453+0000 7f6173fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.454+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860267750 0 0) 0x7f618c1406b0 con 0x7f618c0b1730 2024-03-25T14:24:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.454+0000 7f6173fff640 10 monclient(hunting): _negotiate 2024-03-25T14:24:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.454+0000 7f6173fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.454+0000 7f6173fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.454+0000 7f6173fff640 10 monclient(hunting): my global_id is 4410 2024-03-25T14:24:02.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.454+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618c141a70 con 0x7f618c0b1730 2024-03-25T14:24:02.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.466+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 881664573 0 0) 0x7f6178004eb0 con 0x7f618c0b4d60 2024-03-25T14:24:02.470 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.466+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f618c1406b0 con 0x7f618c0b4d60 2024-03-25T14:24:02.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.468+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6180001bc0 con 0x7f618c0b4d60 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.468+0000 7f6173fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.468+0000 7f6173fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.468+0000 7f6173fff640 10 monclient(hunting): dump: 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:02.472 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.468+0000 7f6173fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.469+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995190655 0 0) 0x7f618c135810 con 0x7f618c14be90 2024-03-25T14:24:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.469+0000 7f6173fff640 10 monclient(hunting): _negotiate 2024-03-25T14:24:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.469+0000 7f6173fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.469+0000 7f6173fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.469+0000 7f6173fff640 10 monclient(hunting): my global_id is 4394 2024-03-25T14:24:02.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.469+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6178004eb0 con 0x7f618c14be90 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.478+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2125224305 0 0) 0x7f6178004eb0 con 0x7f618c14be90 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.478+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f618c135810 con 0x7f618c14be90 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.478+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6184001bd0 con 0x7f618c14be90 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.478+0000 7f6173fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.478+0000 7f6173fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.478+0000 7f6173fff640 10 monclient(hunting): dump: 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.482 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:02.483 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:02.483 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:02.483 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:02.483 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:02.483 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.480+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2168911922 0 0) 0x7f618c1406b0 con 0x7f618c0b4d60 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.480+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 >> v1:172.21.15.63:6790/0 conn(0x7f618c14be90 legacy=0x7f618c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 >> v1:172.21.15.63:6789/0 conn(0x7f618c0b1730 legacy=0x7f618c1daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 10 monclient: _finish_hunting 0 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 1 monclient: found mon.b 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:02.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c140380 con 0x7f618c0b4d60 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 10 monclient: _finish_auth 0 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 10 monclient: _check_auth_tickets 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6173fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 5 monclient: authenticate success, global_id 4414 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 10 monclient: _renew_subs 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c1381d0 con 0x7f618c0b4d60 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 10 monclient: _renew_subs 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c1379a0 con 0x7f618c0b4d60 2024-03-25T14:24:02.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.481+0000 7f6191c81640 1 librados: init done 2024-03-25T14:24:02.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.482+0000 7f6191c81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:02.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.482+0000 7f6191c81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:02.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.482+0000 7f6191c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:02.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.482+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c0627b0 con 0x7f618c0b4d60 2024-03-25T14:24:02.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.753+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6180001220 con 0x7f618c0b4d60 2024-03-25T14:24:02.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.753+0000 7f6173fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:02.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.753+0000 7f6189ffb640 4 set_mon_vals no callback set 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.763+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6180004c30 con 0x7f618c0b4d60 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.763+0000 7f6173fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.763+0000 7f6173fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.763+0000 7f6173fff640 10 monclient: dump: 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:02.767 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:02.768 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:02.768 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:02.768 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:02.768 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:02.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.764+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f618001ac00 con 0x7f618c0b4d60 2024-03-25T14:24:02.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.765+0000 7f618b7fe640 10 monclient: get_auth_request con 0x7f617806dee0 auth_method 0 2024-03-25T14:24:02.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.765+0000 7f6173fff640 10 client.4414.objecter ms_handle_connect 0x7f617806dee0 2024-03-25T14:24:02.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.768+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f6180054920 con 0x7f618c0b4d60 2024-03-25T14:24:02.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.768+0000 7f6173fff640 10 client.4414.objecter ms_dispatch 0x7f618c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:02.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.768+0000 7f6173fff640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:02.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.768+0000 7f6173fff640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:02.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.768+0000 7f6173fff640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T14:24:02.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.772+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6180050260 con 0x7f618c0b4d60 2024-03-25T14:24:02.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.772+0000 7f6173fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:02.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:02.772+0000 7f6173fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:03.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.032+0000 7f6191c81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:03.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.032+0000 7f6191c81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:03.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.032+0000 7f6191c81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:03.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.032+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f618c069f40 con 0x7f618c0b4d60 2024-03-25T14:24:03.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.070+0000 7f6173fff640 1 -- 172.21.15.63:0/2355095433 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f6180052360 con 0x7f618c0b4d60 2024-03-25T14:24:03.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.070+0000 7f6173fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:03.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.070+0000 7f6173fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:03.074 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:03.075 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:03.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 10 librados: watch_flush enter 2024-03-25T14:24:03.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 10 librados: watch_flush exit 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 10 client.4414.objecter successfully canceled tick 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 >> v1:172.21.15.153:6800/1688255552 conn(0x7f617806dee0 legacy=0x7f61780703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 10 monclient: shutdown 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 >> v1:172.21.15.153:6789/0 conn(0x7f618c0b4d60 legacy=0x7f618c1349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 shutdown_connections 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 >> 172.21.15.63:0/2355095433 conn(0x7f618c0617a0 msgr2=0x7f618c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 shutdown_connections 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.073+0000 7f6191c81640 1 -- 172.21.15.63:0/2355095433 wait complete. 2024-03-25T14:24:03.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:03.074+0000 7f6191c81640 1 librados: shutdown 2024-03-25T14:24:03.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:06.517 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:45.030 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.565 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:45.078 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.620 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:04:45.133 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.635 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:45.148 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.641 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:45.101 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.648 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:45.162 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.650 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:45.110 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.675 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:04:45.188 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:06.692 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:45.152 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:09.895 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.005+0000 7f4430bed640 10 monclient: get_monmap_and_config 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.005+0000 7f4430bed640 10 monclient: build_initial_monmap 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.005+0000 7f4430bed640 10 monclient: monmap: 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:10.006444+0000 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:10.006444+0000 2024-03-25T14:24:10.009 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:10.010 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:10.010 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:10.010 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:10.010 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:10.010 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 1 Processor -- start 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 1 -- start start 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 10 monclient: init 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 10 monclient(hunting): picked mon.noname-b con 0x7f442c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 10 monclient(hunting): picked mon.noname-c con 0x7f442c13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 10 monclient(hunting): picked mon.noname-a con 0x7f442c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c142520 con 0x7f442c1430c0 2024-03-25T14:24:10.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.006+0000 7f4430bed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c0b28a0 con 0x7f442c13f710 2024-03-25T14:24:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.007+0000 7f4430bed640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c0b2a30 con 0x7f442c0a98f0 2024-03-25T14:24:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.007+0000 7f4430bed640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:10.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.007+0000 7f4430bed640 10 monclient(hunting): authenticate will time out at 1613.151527s 2024-03-25T14:24:10.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.014+0000 7f442a59c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f442c0a98f0 0x7f442c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:52276/0 (socket says 172.21.15.63:52276) 2024-03-25T14:24:10.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.014+0000 7f442a59c640 1 -- 172.21.15.63:0/1032861005 learned_addr learned my addr 172.21.15.63:0/1032861005 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:10.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.027+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622417515 0 0) 0x7f442c0b28a0 con 0x7f442c13f710 2024-03-25T14:24:10.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.027+0000 7f442959a640 10 monclient(hunting): _negotiate 2024-03-25T14:24:10.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.027+0000 7f442959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:10.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.027+0000 7f442959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:10.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.027+0000 7f442959a640 10 monclient(hunting): my global_id is 4417 2024-03-25T14:24:10.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.027+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4414003520 con 0x7f442c13f710 2024-03-25T14:24:10.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.029+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420121411 0 0) 0x7f442c142520 con 0x7f442c1430c0 2024-03-25T14:24:10.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.029+0000 7f442959a640 10 monclient(hunting): _negotiate 2024-03-25T14:24:10.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.029+0000 7f442959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:10.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.029+0000 7f442959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:10.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.029+0000 7f442959a640 10 monclient(hunting): my global_id is 4413 2024-03-25T14:24:10.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.029+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442c0b28a0 con 0x7f442c1430c0 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194856820 0 0) 0x7f4414003520 con 0x7f442c13f710 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 >> v1:172.21.15.63:6790/0 conn(0x7f442c0a98f0 legacy=0x7f442c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 >> v1:172.21.15.63:6789/0 conn(0x7f442c1430c0 legacy=0x7f442c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 10 monclient: _finish_hunting 0 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 1 monclient: found mon.noname-c 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c0b44c0 con 0x7f442c13f710 2024-03-25T14:24:10.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 10 monclient: _finish_auth 0 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 10 monclient: _check_auth_tickets 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4420001690 con 0x7f442c13f710 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f442959a640 10 monclient: dump: 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:10.040 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:10.041 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:10.041 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:10.041 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:10.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f4430bed640 5 monclient: authenticate success, global_id 4417 2024-03-25T14:24:10.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.035+0000 7f4430bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44200017f0 con 0x7f442c13f710 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f442959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f442959a640 1 -- 172.21.15.63:0/1032861005 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4420001c00 con 0x7f442c13f710 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f4430bed640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f4430bed640 4 set_mon_vals no callback set 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f442959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f442959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:10.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.051+0000 7f442959a640 10 monclient: dump: 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:10.056 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:10.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.052+0000 7f4430bed640 10 monclient: shutdown 2024-03-25T14:24:10.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.052+0000 7f4430bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.052+0000 7f4430bed640 1 -- 172.21.15.63:0/1032861005 >> v1:172.21.15.153:6789/0 conn(0x7f442c13f710 legacy=0x7f442c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.052+0000 7f4430bed640 1 -- 172.21.15.63:0/1032861005 shutdown_connections 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.052+0000 7f4430bed640 1 -- 172.21.15.63:0/1032861005 >> 172.21.15.63:0/1032861005 conn(0x7f442c0b7ea0 msgr2=0x7f442c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.052+0000 7f4430bed640 1 -- 172.21.15.63:0/1032861005 shutdown_connections 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.052+0000 7f4430bed640 1 -- 172.21.15.63:0/1032861005 wait complete. 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.053+0000 7f4430bed640 10 monclient: build_initial_monmap 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.053+0000 7f4430bed640 10 monclient: monmap: 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:10.059 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 librados: starting msgr at 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 librados: starting objecter 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 Processor -- start 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 -- start start 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 librados: setting wanted keys 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 librados: calling monclient init 2024-03-25T14:24:10.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient: init 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient(hunting): picked mon.noname-a con 0x7f442c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient(hunting): picked mon.noname-c con 0x7f442c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient(hunting): picked mon.noname-b con 0x7f442c1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c1d7490 con 0x7f442c0a98f0 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c1d76c0 con 0x7f442c1430c0 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c1d82f0 con 0x7f442c13f710 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:10.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.054+0000 7f4430bed640 10 monclient(hunting): authenticate will time out at 1613.199399s 2024-03-25T14:24:10.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.065+0000 7f442a59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f442c0a98f0 0x7f442c1d7be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54590/0 (socket says 172.21.15.63:54590) 2024-03-25T14:24:10.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.065+0000 7f442a59c640 1 -- 172.21.15.63:0/2574607106 learned_addr learned my addr 172.21.15.63:0/2574607106 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:10.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.067+0000 7f441affd640 10 client.?.objecter ms_handle_connect 0x7f442c1430c0 2024-03-25T14:24:10.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.067+0000 7f441affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:10.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.068+0000 7f441affd640 10 client.?.objecter ms_handle_connect 0x7f442c0a98f0 2024-03-25T14:24:10.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.068+0000 7f441affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:10.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.076+0000 7f441affd640 10 client.?.objecter ms_handle_connect 0x7f442c13f710 2024-03-25T14:24:10.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.076+0000 7f441affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:10.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.078+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764764430 0 0) 0x7f442c1d76c0 con 0x7f442c1430c0 2024-03-25T14:24:10.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.078+0000 7f441affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:10.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.078+0000 7f441affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:10.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.078+0000 7f441affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:10.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.078+0000 7f441affd640 10 monclient(hunting): my global_id is 4420 2024-03-25T14:24:10.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.078+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f440c004eb0 con 0x7f442c1430c0 2024-03-25T14:24:10.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.079+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820971734 0 0) 0x7f442c1d7490 con 0x7f442c0a98f0 2024-03-25T14:24:10.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.079+0000 7f441affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:10.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.079+0000 7f441affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:10.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.079+0000 7f441affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:10.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.079+0000 7f441affd640 10 monclient(hunting): my global_id is 4416 2024-03-25T14:24:10.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.079+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442c1d76c0 con 0x7f442c0a98f0 2024-03-25T14:24:10.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.094+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157552376 0 0) 0x7f442c1d82f0 con 0x7f442c13f710 2024-03-25T14:24:10.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.094+0000 7f441affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:10.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.094+0000 7f441affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:10.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.094+0000 7f441affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:10.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.094+0000 7f441affd640 10 monclient(hunting): my global_id is 4400 2024-03-25T14:24:10.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.094+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442c1d7490 con 0x7f442c13f710 2024-03-25T14:24:10.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.095+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704638738 0 0) 0x7f440c004eb0 con 0x7f442c1430c0 2024-03-25T14:24:10.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.095+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f442c1d82f0 con 0x7f442c1430c0 2024-03-25T14:24:10.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.095+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4424001bd0 con 0x7f442c1430c0 2024-03-25T14:24:10.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.095+0000 7f441affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:10.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.095+0000 7f441affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:10.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.095+0000 7f441affd640 10 monclient(hunting): dump: 2024-03-25T14:24:10.099 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:10.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.095+0000 7f441affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:10.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.105+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1034069662 0 0) 0x7f442c1d76c0 con 0x7f442c0a98f0 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.105+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f440c004eb0 con 0x7f442c0a98f0 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.105+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f441c0018f0 con 0x7f442c0a98f0 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.105+0000 7f441affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.105+0000 7f441affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.105+0000 7f441affd640 10 monclient(hunting): dump: 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:10.109 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 91238383 0 0) 0x7f442c1d82f0 con 0x7f442c1430c0 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 >> v1:172.21.15.63:6790/0 conn(0x7f442c13f710 legacy=0x7f442c1383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 >> v1:172.21.15.63:6789/0 conn(0x7f442c0a98f0 legacy=0x7f442c1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 10 monclient: _finish_hunting 0 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 1 monclient: found mon.b 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c138af0 con 0x7f442c1430c0 2024-03-25T14:24:10.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 10 monclient: _finish_auth 0 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 10 monclient: _check_auth_tickets 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f441affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 5 monclient: authenticate success, global_id 4420 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 10 monclient: _renew_subs 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442c1376d0 con 0x7f442c1430c0 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 10 monclient: _renew_subs 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c134fc0 con 0x7f442c1430c0 2024-03-25T14:24:10.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.116+0000 7f4430bed640 1 librados: init done 2024-03-25T14:24:10.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.117+0000 7f4430bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:10.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.117+0000 7f4430bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:10.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.117+0000 7f4430bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:10.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.117+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442c0627b0 con 0x7f442c1430c0 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.422+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44240020c0 con 0x7f442c1430c0 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.422+0000 7f441affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.422+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4424003b10 con 0x7f442c1430c0 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.422+0000 7f441affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.422+0000 7f441affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.422+0000 7f441affd640 10 monclient: dump: 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:10.426 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:10.427 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:10.427 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:10.427 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:10.427 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:10.427 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:10.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.422+0000 7f4428d99640 4 set_mon_vals no callback set 2024-03-25T14:24:10.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.423+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f4424019ba0 con 0x7f442c1430c0 2024-03-25T14:24:10.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.423+0000 7f442a59c640 10 monclient: get_auth_request con 0x7f440c06df70 auth_method 0 2024-03-25T14:24:10.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.423+0000 7f441affd640 10 client.4420.objecter ms_handle_connect 0x7f440c06df70 2024-03-25T14:24:10.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.427+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f44240538b0 con 0x7f442c1430c0 2024-03-25T14:24:10.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.427+0000 7f441affd640 10 client.4420.objecter ms_dispatch 0x7f442c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:10.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.427+0000 7f441affd640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:10.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.427+0000 7f441affd640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:10.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.427+0000 7f441affd640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T14:24:10.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.428+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f442404f1f0 con 0x7f442c1430c0 2024-03-25T14:24:10.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.428+0000 7f441affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:10.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.428+0000 7f441affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:10.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.672+0000 7f4430bed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:10.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.672+0000 7f4430bed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:10.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.672+0000 7f4430bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:10.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.672+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f442c14f300 con 0x7f442c1430c0 2024-03-25T14:24:10.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.707+0000 7f441affd640 1 -- 172.21.15.63:0/2574607106 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f44240512f0 con 0x7f442c1430c0 2024-03-25T14:24:10.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.707+0000 7f441affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:10.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.707+0000 7f441affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:10.711 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:10.712 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:10.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 10 librados: watch_flush enter 2024-03-25T14:24:10.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 10 librados: watch_flush exit 2024-03-25T14:24:10.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T14:24:10.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 10 client.4420.objecter successfully canceled tick 2024-03-25T14:24:10.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 >> v1:172.21.15.153:6800/1688255552 conn(0x7f440c06df70 legacy=0x7f440c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:10.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 10 monclient: shutdown 2024-03-25T14:24:10.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:10.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 >> v1:172.21.15.153:6789/0 conn(0x7f442c1430c0 legacy=0x7f442c133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:10.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 shutdown_connections 2024-03-25T14:24:10.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 >> 172.21.15.63:0/2574607106 conn(0x7f442c0b7ea0 msgr2=0x7f442c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:10.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 shutdown_connections 2024-03-25T14:24:10.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 1 -- 172.21.15.63:0/2574607106 wait complete. 2024-03-25T14:24:10.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:10.710+0000 7f4430bed640 1 librados: shutdown 2024-03-25T14:24:10.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:17.532 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:17.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.645+0000 7ff568f8c640 10 monclient: get_monmap_and_config 2024-03-25T14:24:17.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.645+0000 7ff568f8c640 10 monclient: build_initial_monmap 2024-03-25T14:24:17.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.645+0000 7ff568f8c640 10 monclient: monmap: 2024-03-25T14:24:17.649 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:17.649 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:17.646812+0000 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:17.646812+0000 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:17.650 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:17.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.646+0000 7ff568f8c640 1 Processor -- start 2024-03-25T14:24:17.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.646+0000 7ff568f8c640 1 -- start start 2024-03-25T14:24:17.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.646+0000 7ff568f8c640 10 monclient: init 2024-03-25T14:24:17.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:17.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:24:17.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff5640a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:17.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff56413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:17.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff5641430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:17.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff564142520 con 0x7ff5641430c0 2024-03-25T14:24:17.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640b28a0 con 0x7ff5640a98f0 2024-03-25T14:24:17.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5640b2a30 con 0x7ff56413f710 2024-03-25T14:24:17.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:17.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.647+0000 7ff568f8c640 10 monclient(hunting): authenticate will time out at 1620.791894s 2024-03-25T14:24:17.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.655+0000 7ff56259c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff5640a98f0 0x7ff5640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:45806/0 (socket says 172.21.15.63:45806) 2024-03-25T14:24:17.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.655+0000 7ff56259c640 1 -- 172.21.15.63:0/2335133681 learned_addr learned my addr 172.21.15.63:0/2335133681 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:17.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.669+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109868145 0 0) 0x7ff5640b2a30 con 0x7ff56413f710 2024-03-25T14:24:17.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.669+0000 7ff56159a640 10 monclient(hunting): _negotiate 2024-03-25T14:24:17.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.669+0000 7ff56159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:17.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.669+0000 7ff56159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:17.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.669+0000 7ff56159a640 10 monclient(hunting): my global_id is 4403 2024-03-25T14:24:17.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.669+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff54c003520 con 0x7ff56413f710 2024-03-25T14:24:17.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.672+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126839466 0 0) 0x7ff5640b28a0 con 0x7ff5640a98f0 2024-03-25T14:24:17.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.672+0000 7ff56159a640 10 monclient(hunting): _negotiate 2024-03-25T14:24:17.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.672+0000 7ff56159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:17.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.672+0000 7ff56159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:17.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.672+0000 7ff56159a640 10 monclient(hunting): my global_id is 4423 2024-03-25T14:24:17.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.672+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5640b2a30 con 0x7ff5640a98f0 2024-03-25T14:24:17.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3398997684 0 0) 0x7ff5640b2a30 con 0x7ff5640a98f0 2024-03-25T14:24:17.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 >> v1:172.21.15.63:6790/0 conn(0x7ff56413f710 legacy=0x7ff5641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:17.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 >> v1:172.21.15.63:6789/0 conn(0x7ff5641430c0 legacy=0x7ff56414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 10 monclient: _finish_hunting 0 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 1 monclient: found mon.noname-c 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5640b44c0 con 0x7ff5640a98f0 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 10 monclient: _finish_auth 0 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 10 monclient: _check_auth_tickets 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff554001670 con 0x7ff5640a98f0 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.680+0000 7ff56159a640 10 monclient: dump: 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:17.685 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.681+0000 7ff568f8c640 5 monclient: authenticate success, global_id 4423 2024-03-25T14:24:17.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.681+0000 7ff568f8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5540017d0 con 0x7ff5640a98f0 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff56159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff56159a640 1 -- 172.21.15.63:0/2335133681 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff554001ba0 con 0x7ff5640a98f0 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff56159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff56159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff56159a640 10 monclient: dump: 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:17.699 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff568f8c640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:17.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.695+0000 7ff568f8c640 4 set_mon_vals no callback set 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.696+0000 7ff568f8c640 10 monclient: shutdown 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.696+0000 7ff568f8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.696+0000 7ff568f8c640 1 -- 172.21.15.63:0/2335133681 >> v1:172.21.15.153:6789/0 conn(0x7ff5640a98f0 legacy=0x7ff5640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.696+0000 7ff568f8c640 1 -- 172.21.15.63:0/2335133681 shutdown_connections 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.696+0000 7ff568f8c640 1 -- 172.21.15.63:0/2335133681 >> 172.21.15.63:0/2335133681 conn(0x7ff5640b7ea0 msgr2=0x7ff5640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.696+0000 7ff568f8c640 1 -- 172.21.15.63:0/2335133681 shutdown_connections 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.696+0000 7ff568f8c640 1 -- 172.21.15.63:0/2335133681 wait complete. 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.697+0000 7ff568f8c640 10 monclient: build_initial_monmap 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.697+0000 7ff568f8c640 10 monclient: monmap: 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:17.702 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.698+0000 7ff568f8c640 1 librados: starting msgr at 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.698+0000 7ff568f8c640 1 librados: starting objecter 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.698+0000 7ff568f8c640 1 Processor -- start 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.698+0000 7ff568f8c640 1 -- start start 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.698+0000 7ff568f8c640 1 librados: setting wanted keys 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.698+0000 7ff568f8c640 1 librados: calling monclient init 2024-03-25T14:24:17.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.698+0000 7ff568f8c640 10 monclient: init 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff5640a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff56413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff5641430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5641dabe0 con 0x7ff56413f710 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5641dbab0 con 0x7ff5641430c0 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5641d5060 con 0x7ff5640a98f0 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:17.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.699+0000 7ff568f8c640 10 monclient(hunting): authenticate will time out at 1620.844054s 2024-03-25T14:24:17.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.704+0000 7ff562d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff5641430c0 0x7ff564138460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:45850/0 (socket says 172.21.15.63:45850) 2024-03-25T14:24:17.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.704+0000 7ff562d9d640 1 -- 172.21.15.63:0/763723286 learned_addr learned my addr 172.21.15.63:0/763723286 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:17.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.707+0000 7ff552ffd640 10 client.?.objecter ms_handle_connect 0x7ff5641430c0 2024-03-25T14:24:17.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.707+0000 7ff552ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:17.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.713+0000 7ff552ffd640 10 client.?.objecter ms_handle_connect 0x7ff56413f710 2024-03-25T14:24:17.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.713+0000 7ff552ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:17.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.719+0000 7ff552ffd640 10 client.?.objecter ms_handle_connect 0x7ff5640a98f0 2024-03-25T14:24:17.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.719+0000 7ff552ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:17.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.727+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930731365 0 0) 0x7ff5641dabe0 con 0x7ff56413f710 2024-03-25T14:24:17.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.727+0000 7ff552ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:17.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.727+0000 7ff552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:17.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.727+0000 7ff552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:17.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.727+0000 7ff552ffd640 10 monclient(hunting): my global_id is 4422 2024-03-25T14:24:17.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.727+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff544004eb0 con 0x7ff56413f710 2024-03-25T14:24:17.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251381 0 0) 0x7ff5641dbab0 con 0x7ff5641430c0 2024-03-25T14:24:17.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): my global_id is 4426 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5641dabe0 con 0x7ff5641430c0 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402964907 0 0) 0x7ff5641d5060 con 0x7ff5640a98f0 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 10 monclient(hunting): my global_id is 4406 2024-03-25T14:24:17.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.730+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5641dbab0 con 0x7ff5640a98f0 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.740+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502140934 0 0) 0x7ff5641dabe0 con 0x7ff5641430c0 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.740+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5641d5060 con 0x7ff5641430c0 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.740+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff55c001bd0 con 0x7ff5641430c0 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.740+0000 7ff552ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.740+0000 7ff552ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.740+0000 7ff552ffd640 10 monclient(hunting): dump: 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:17.744 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:17.745 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:17.745 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:17.745 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:17.745 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:17.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.740+0000 7ff552ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.745+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2334504148 0 0) 0x7ff5641dbab0 con 0x7ff5640a98f0 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.745+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5641dabe0 con 0x7ff5640a98f0 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.745+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff554001ae0 con 0x7ff5640a98f0 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.745+0000 7ff552ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.745+0000 7ff552ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.745+0000 7ff552ffd640 10 monclient(hunting): dump: 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:17.749 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.750 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.750 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:17.750 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:17.750 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:17.750 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:17.750 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:17.750 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.750+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2784832173 0 0) 0x7ff544004eb0 con 0x7ff56413f710 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.750+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5641dbab0 con 0x7ff56413f710 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.750+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff558001950 con 0x7ff56413f710 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.750+0000 7ff552ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.750+0000 7ff552ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.750+0000 7ff552ffd640 10 monclient(hunting): dump: 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:17.755 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 347582595 0 0) 0x7ff5641dabe0 con 0x7ff5640a98f0 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 >> v1:172.21.15.153:6789/0 conn(0x7ff5641430c0 legacy=0x7ff564138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 >> v1:172.21.15.63:6789/0 conn(0x7ff56413f710 legacy=0x7ff5641d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:24:17.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 1 monclient: found mon.c 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5641d51f0 con 0x7ff5640a98f0 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 10 monclient: _finish_auth 0 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff552ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 5 monclient: authenticate success, global_id 4406 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 10 monclient: _renew_subs 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff564137fa0 con 0x7ff5640a98f0 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 10 monclient: _renew_subs 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:17.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5641d41e0 con 0x7ff5640a98f0 2024-03-25T14:24:17.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.751+0000 7ff568f8c640 1 librados: init done 2024-03-25T14:24:17.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.752+0000 7ff568f8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:17.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.752+0000 7ff568f8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:17.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.752+0000 7ff568f8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:17.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:17.752+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff564074cc0 con 0x7ff5640a98f0 2024-03-25T14:24:18.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff554004a90 con 0x7ff5640a98f0 2024-03-25T14:24:18.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff552ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:18.067 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff554001c60 con 0x7ff5640a98f0 2024-03-25T14:24:18.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff552ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:18.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff552ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:18.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff552ffd640 10 monclient: dump: 2024-03-25T14:24:18.068 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:18.068 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:18.068 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:18.069 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:18.069 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:18.069 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:18.069 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:18.069 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:18.069 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:18.070 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:18.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff560d99640 4 set_mon_vals no callback set 2024-03-25T14:24:18.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.063+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7ff55401ac10 con 0x7ff5640a98f0 2024-03-25T14:24:18.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.064+0000 7ff561d9b640 10 monclient: get_auth_request con 0x7ff54406e160 auth_method 0 2024-03-25T14:24:18.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.064+0000 7ff552ffd640 10 client.4406.objecter ms_handle_connect 0x7ff54406e160 2024-03-25T14:24:18.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.069+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7ff554054820 con 0x7ff5640a98f0 2024-03-25T14:24:18.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.069+0000 7ff552ffd640 10 client.4406.objecter ms_dispatch 0x7ff564000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:18.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.069+0000 7ff552ffd640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:18.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.069+0000 7ff552ffd640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:18.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.069+0000 7ff552ffd640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-25T14:24:18.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.070+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff554050330 con 0x7ff5640a98f0 2024-03-25T14:24:18.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.070+0000 7ff552ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:18.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.070+0000 7ff552ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:18.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.317+0000 7ff568f8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:18.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.317+0000 7ff568f8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:18.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.317+0000 7ff568f8c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:18.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.317+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5640627b0 con 0x7ff5640a98f0 2024-03-25T14:24:18.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.445+0000 7ff552ffd640 1 -- 172.21.15.63:0/763723286 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7ff554052260 con 0x7ff5640a98f0 2024-03-25T14:24:18.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.445+0000 7ff552ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:18.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.445+0000 7ff552ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:18.450 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:18.450 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.447+0000 7ff568f8c640 10 librados: watch_flush enter 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 10 librados: watch_flush exit 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 10 client.4406.objecter successfully canceled tick 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 >> v1:172.21.15.153:6800/1688255552 conn(0x7ff54406e160 legacy=0x7ff544070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 10 monclient: shutdown 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 >> v1:172.21.15.63:6790/0 conn(0x7ff5640a98f0 legacy=0x7ff5641db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 shutdown_connections 2024-03-25T14:24:18.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 >> 172.21.15.63:0/763723286 conn(0x7ff5640b7ea0 msgr2=0x7ff564074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:18.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 shutdown_connections 2024-03-25T14:24:18.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 1 -- 172.21.15.63:0/763723286 wait complete. 2024-03-25T14:24:18.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:18.448+0000 7ff568f8c640 1 librados: shutdown 2024-03-25T14:24:18.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:25.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:25.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.372+0000 7f1954f59640 10 monclient: get_monmap_and_config 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.372+0000 7f1954f59640 10 monclient: build_initial_monmap 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.372+0000 7f1954f59640 10 monclient: monmap: 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:25.373284+0000 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:25.373284+0000 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:25.376 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:25.377 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.373+0000 7f1954f59640 1 Processor -- start 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.373+0000 7f1954f59640 1 -- start start 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.373+0000 7f1954f59640 10 monclient: init 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.373+0000 7f1954f59640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.373+0000 7f1954f59640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.373+0000 7f1954f59640 10 monclient(hunting): picked mon.noname-a con 0x7f19500a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.373+0000 7f1954f59640 10 monclient(hunting): picked mon.noname-c con 0x7f195013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.374+0000 7f1954f59640 10 monclient(hunting): picked mon.noname-b con 0x7f19501430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.374+0000 7f1954f59640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1950142520 con 0x7f19500a98f0 2024-03-25T14:24:25.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.374+0000 7f1954f59640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500b28a0 con 0x7f195013f710 2024-03-25T14:24:25.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.374+0000 7f1954f59640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500b2a30 con 0x7f19501430c0 2024-03-25T14:24:25.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.374+0000 7f1954f59640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:25.379 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.374+0000 7f1954f59640 10 monclient(hunting): authenticate will time out at 1628.518738s 2024-03-25T14:24:25.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.389+0000 7f194ed9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f19501430c0 0x7f195014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:51898/0 (socket says 172.21.15.63:51898) 2024-03-25T14:24:25.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.389+0000 7f194ed9d640 1 -- 172.21.15.63:0/1715203036 learned_addr learned my addr 172.21.15.63:0/1715203036 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:25.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.404+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713934721 0 0) 0x7f19500b2a30 con 0x7f19501430c0 2024-03-25T14:24:25.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.404+0000 7f194dd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:24:25.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.404+0000 7f194dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:25.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.404+0000 7f194dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:25.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.404+0000 7f194dd9b640 10 monclient(hunting): my global_id is 4409 2024-03-25T14:24:25.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.404+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f193c003520 con 0x7f19501430c0 2024-03-25T14:24:25.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.407+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954899596 0 0) 0x7f19500b28a0 con 0x7f195013f710 2024-03-25T14:24:25.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.407+0000 7f194dd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:24:25.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.407+0000 7f194dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:25.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.407+0000 7f194dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:25.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.407+0000 7f194dd9b640 10 monclient(hunting): my global_id is 4429 2024-03-25T14:24:25.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.407+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19500b2a30 con 0x7f195013f710 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067465001 0 0) 0x7f19500b2a30 con 0x7f195013f710 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 >> v1:172.21.15.63:6790/0 conn(0x7f19501430c0 legacy=0x7f195014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 >> v1:172.21.15.63:6789/0 conn(0x7f19500a98f0 legacy=0x7f19500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 10 monclient: _finish_hunting 0 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 1 monclient: found mon.noname-c 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500b44c0 con 0x7f195013f710 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 10 monclient: _finish_auth 0 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 10 monclient: _check_auth_tickets 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:25.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1938001670 con 0x7f195013f710 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f194dd9b640 10 monclient: dump: 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:25.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f1954f59640 5 monclient: authenticate success, global_id 4429 2024-03-25T14:24:25.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.419+0000 7f1954f59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:25.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19380017d0 con 0x7f195013f710 2024-03-25T14:24:25.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f194dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:25.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f194dd9b640 1 -- 172.21.15.63:0/1715203036 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1938001ba0 con 0x7f195013f710 2024-03-25T14:24:25.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f194dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:25.442 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f194dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f194dd9b640 10 monclient: dump: 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f1954f59640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:25.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.438+0000 7f1954f59640 4 set_mon_vals no callback set 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.439+0000 7f1954f59640 10 monclient: shutdown 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.439+0000 7f1954f59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.439+0000 7f1954f59640 1 -- 172.21.15.63:0/1715203036 >> v1:172.21.15.153:6789/0 conn(0x7f195013f710 legacy=0x7f19501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.439+0000 7f1954f59640 1 -- 172.21.15.63:0/1715203036 shutdown_connections 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.439+0000 7f1954f59640 1 -- 172.21.15.63:0/1715203036 >> 172.21.15.63:0/1715203036 conn(0x7f19500b7ea0 msgr2=0x7f19500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.439+0000 7f1954f59640 1 -- 172.21.15.63:0/1715203036 shutdown_connections 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.439+0000 7f1954f59640 1 -- 172.21.15.63:0/1715203036 wait complete. 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.440+0000 7f1954f59640 10 monclient: build_initial_monmap 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.440+0000 7f1954f59640 10 monclient: monmap: 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:25.446 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 librados: starting msgr at 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 librados: starting objecter 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 Processor -- start 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 -- start start 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 librados: setting wanted keys 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 librados: calling monclient init 2024-03-25T14:24:25.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient: init 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient(hunting): picked mon.noname-a con 0x7f19500a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient(hunting): picked mon.noname-b con 0x7f195013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient(hunting): picked mon.noname-c con 0x7f19501430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500b1610 con 0x7f19500a98f0 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19500b0b50 con 0x7f195013f710 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19501d7b70 con 0x7f19501430c0 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:25.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.441+0000 7f1954f59640 10 monclient(hunting): authenticate will time out at 1628.586413s 2024-03-25T14:24:25.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.457+0000 7f1947fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f195013f710 0x7f19501d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:54140/0 (socket says 172.21.15.63:54140) 2024-03-25T14:24:25.461 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.457+0000 7f1947fff640 1 -- 172.21.15.63:0/2342101408 learned_addr learned my addr 172.21.15.63:0/2342101408 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:25.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.459+0000 7f1946ffd640 10 client.?.objecter ms_handle_connect 0x7f195013f710 2024-03-25T14:24:25.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.459+0000 7f1946ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:25.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.464+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766169934 0 0) 0x7f19500b0b50 con 0x7f195013f710 2024-03-25T14:24:25.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.464+0000 7f1946ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:25.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.464+0000 7f1946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:25.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.464+0000 7f1946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:25.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.464+0000 7f1946ffd640 10 monclient(hunting): my global_id is 4432 2024-03-25T14:24:25.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.464+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1930004eb0 con 0x7f195013f710 2024-03-25T14:24:25.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.465+0000 7f1946ffd640 10 client.?.objecter ms_handle_connect 0x7f19501430c0 2024-03-25T14:24:25.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.465+0000 7f1946ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:25.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.469+0000 7f1946ffd640 10 client.?.objecter ms_handle_connect 0x7f19500a98f0 2024-03-25T14:24:25.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.469+0000 7f1946ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:25.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.475+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306038574 0 0) 0x7f19501d7b70 con 0x7f19501430c0 2024-03-25T14:24:25.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.475+0000 7f1946ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:25.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.475+0000 7f1946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:25.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.475+0000 7f1946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:25.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.475+0000 7f1946ffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T14:24:25.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.475+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19500b0b50 con 0x7f19501430c0 2024-03-25T14:24:25.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.480+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321535715 0 0) 0x7f1930004eb0 con 0x7f195013f710 2024-03-25T14:24:25.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.480+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19501d7b70 con 0x7f195013f710 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.480+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1938001be0 con 0x7f195013f710 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.480+0000 7f1946ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.480+0000 7f1946ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.480+0000 7f1946ffd640 10 monclient(hunting): dump: 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:25.484 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:25.485 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:25.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.480+0000 7f1946ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:25.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.492+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230712636 0 0) 0x7f19500b1610 con 0x7f19500a98f0 2024-03-25T14:24:25.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.492+0000 7f1946ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:25.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.492+0000 7f1946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:25.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.492+0000 7f1946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:25.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.492+0000 7f1946ffd640 10 monclient(hunting): my global_id is 4428 2024-03-25T14:24:25.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.492+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1930004eb0 con 0x7f19500a98f0 2024-03-25T14:24:25.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 188210265 0 0) 0x7f19501d7b70 con 0x7f195013f710 2024-03-25T14:24:25.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 >> v1:172.21.15.63:6790/0 conn(0x7f19501430c0 legacy=0x7f19501d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:25.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 >> v1:172.21.15.63:6789/0 conn(0x7f19500a98f0 legacy=0x7f19500b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:25.501 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 1 monclient: found mon.b 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19501d5000 con 0x7f195013f710 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 10 monclient: _finish_auth 0 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 5 monclient: authenticate success, global_id 4432 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 10 monclient: _renew_subs 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19501d51f0 con 0x7f195013f710 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 10 monclient: _renew_subs 2024-03-25T14:24:25.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:25.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19501d7840 con 0x7f195013f710 2024-03-25T14:24:25.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.497+0000 7f1954f59640 1 librados: init done 2024-03-25T14:24:25.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.498+0000 7f1954f59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:25.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.498+0000 7f1954f59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:25.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.498+0000 7f1954f59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:25.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.498+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19500627b0 con 0x7f195013f710 2024-03-25T14:24:25.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.792+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19380038a0 con 0x7f195013f710 2024-03-25T14:24:25.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.792+0000 7f1946ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:25.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.792+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1938003bf0 con 0x7f195013f710 2024-03-25T14:24:25.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.792+0000 7f1946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:25.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.792+0000 7f1946ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:25.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.792+0000 7f1946ffd640 10 monclient: dump: 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.792+0000 7f194d59a640 4 set_mon_vals no callback set 2024-03-25T14:24:25.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.793+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f1938003e10 con 0x7f195013f710 2024-03-25T14:24:25.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.794+0000 7f194e59c640 10 monclient: get_auth_request con 0x7f193006de00 auth_method 0 2024-03-25T14:24:25.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.794+0000 7f1946ffd640 10 client.4432.objecter ms_handle_connect 0x7f193006de00 2024-03-25T14:24:25.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.798+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f1938054930 con 0x7f195013f710 2024-03-25T14:24:25.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.798+0000 7f1946ffd640 10 client.4432.objecter ms_dispatch 0x7f1950000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:25.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.798+0000 7f1946ffd640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:25.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.798+0000 7f1946ffd640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:25.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.798+0000 7f1946ffd640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T14:24:25.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.799+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1938050440 con 0x7f195013f710 2024-03-25T14:24:25.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.799+0000 7f1946ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:25.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:25.799+0000 7f1946ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:26.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.053+0000 7f1954f59640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:26.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.053+0000 7f1954f59640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:26.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.053+0000 7f1954f59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:26.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.053+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f195013c750 con 0x7f195013f710 2024-03-25T14:24:26.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.083+0000 7f1946ffd640 1 -- 172.21.15.63:0/2342101408 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f1938052370 con 0x7f195013f710 2024-03-25T14:24:26.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.083+0000 7f1946ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:26.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.083+0000 7f1946ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:26.087 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:26.088 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 10 librados: watch_flush enter 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 10 librados: watch_flush exit 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 10 client.4432.objecter successfully canceled tick 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 >> v1:172.21.15.153:6800/1688255552 conn(0x7f193006de00 legacy=0x7f19300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 10 monclient: shutdown 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 >> v1:172.21.15.153:6789/0 conn(0x7f195013f710 legacy=0x7f19501d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 shutdown_connections 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.085+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 >> 172.21.15.63:0/2342101408 conn(0x7f19500b7ea0 msgr2=0x7f1950074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.086+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 shutdown_connections 2024-03-25T14:24:26.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.086+0000 7f1954f59640 1 -- 172.21.15.63:0/2342101408 wait complete. 2024-03-25T14:24:26.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:26.086+0000 7f1954f59640 1 librados: shutdown 2024-03-25T14:24:26.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:26.697 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:05.210 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.710 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:05.223 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.749 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:05.262 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.760 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:05.273 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.769 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:05.229 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.776 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:05.236 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.779 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:05.292 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.801 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:05.261 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:26.803 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:05.316 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:32.906 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.013+0000 7f1577e0f640 10 monclient: get_monmap_and_config 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.013+0000 7f1577e0f640 10 monclient: build_initial_monmap 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.013+0000 7f1577e0f640 10 monclient: monmap: 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:33.014367+0000 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:33.014367+0000 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:33.017 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:33.018 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:33.018 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:33.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.014+0000 7f1577e0f640 1 Processor -- start 2024-03-25T14:24:33.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.014+0000 7f1577e0f640 1 -- start start 2024-03-25T14:24:33.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.014+0000 7f1577e0f640 10 monclient: init 2024-03-25T14:24:33.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:33.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:24:33.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f15700a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:33.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f157013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:33.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f15701430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:33.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1570142520 con 0x7f15701430c0 2024-03-25T14:24:33.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700b28a0 con 0x7f15700a98f0 2024-03-25T14:24:33.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700b2a30 con 0x7f157013f710 2024-03-25T14:24:33.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:33.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.015+0000 7f1577e0f640 10 monclient(hunting): authenticate will time out at 1636.159947s 2024-03-25T14:24:33.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.022+0000 7f1575bab640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f15700a98f0 0x7f15700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:58074/0 (socket says 172.21.15.63:58074) 2024-03-25T14:24:33.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.022+0000 7f1575bab640 1 -- 172.21.15.63:0/3575150882 learned_addr learned my addr 172.21.15.63:0/3575150882 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:33.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.041+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995764147 0 0) 0x7f1570142520 con 0x7f15701430c0 2024-03-25T14:24:33.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.041+0000 7f1574ba9640 10 monclient(hunting): _negotiate 2024-03-25T14:24:33.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.041+0000 7f1574ba9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:33.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.041+0000 7f1574ba9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:33.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.041+0000 7f1574ba9640 10 monclient(hunting): my global_id is 4431 2024-03-25T14:24:33.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.041+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f155c003520 con 0x7f15701430c0 2024-03-25T14:24:33.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.044+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827699348 0 0) 0x7f15700b28a0 con 0x7f15700a98f0 2024-03-25T14:24:33.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.044+0000 7f1574ba9640 10 monclient(hunting): _negotiate 2024-03-25T14:24:33.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.044+0000 7f1574ba9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:33.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.044+0000 7f1574ba9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:33.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.044+0000 7f1574ba9640 10 monclient(hunting): my global_id is 4435 2024-03-25T14:24:33.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.044+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1570142520 con 0x7f15700a98f0 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354593593 0 0) 0x7f1570142520 con 0x7f15700a98f0 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 >> v1:172.21.15.63:6790/0 conn(0x7f157013f710 legacy=0x7f15701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 >> v1:172.21.15.63:6789/0 conn(0x7f15701430c0 legacy=0x7f157014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 10 monclient: _finish_hunting 0 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 1 monclient: found mon.noname-c 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:24:33.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15700b44c0 con 0x7f15700a98f0 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 10 monclient: _finish_auth 0 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 10 monclient: _check_auth_tickets 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f156c0019f0 con 0x7f15700a98f0 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1577e0f640 5 monclient: authenticate success, global_id 4435 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1577e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1574ba9640 10 monclient: dump: 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.056 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:33.057 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:33.057 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:33.057 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:33.057 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:33.057 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:33.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.051+0000 7f1577e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f156c001b50 con 0x7f15700a98f0 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1574ba9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1574ba9640 1 -- 172.21.15.63:0/3575150882 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f156c001f20 con 0x7f15700a98f0 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1577e0f640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1577e0f640 4 set_mon_vals no callback set 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1574ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1574ba9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.069+0000 7f1574ba9640 10 monclient: dump: 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:33.073 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.074 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.074 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:33.074 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:33.074 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:33.074 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:33.074 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:33.074 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:33.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.070+0000 7f1577e0f640 10 monclient: shutdown 2024-03-25T14:24:33.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.070+0000 7f1577e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:33.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.070+0000 7f1577e0f640 1 -- 172.21.15.63:0/3575150882 >> v1:172.21.15.153:6789/0 conn(0x7f15700a98f0 legacy=0x7f15700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:33.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.070+0000 7f1577e0f640 1 -- 172.21.15.63:0/3575150882 shutdown_connections 2024-03-25T14:24:33.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.070+0000 7f1577e0f640 1 -- 172.21.15.63:0/3575150882 >> 172.21.15.63:0/3575150882 conn(0x7f15700b7ea0 msgr2=0x7f15700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:33.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.070+0000 7f1577e0f640 1 -- 172.21.15.63:0/3575150882 shutdown_connections 2024-03-25T14:24:33.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.070+0000 7f1577e0f640 1 -- 172.21.15.63:0/3575150882 wait complete. 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 10 monclient: build_initial_monmap 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 10 monclient: monmap: 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:33.076 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 1 librados: starting msgr at 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 1 librados: starting objecter 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 1 Processor -- start 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 1 -- start start 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 1 librados: setting wanted keys 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 1 librados: calling monclient init 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.071+0000 7f1577e0f640 10 monclient: init 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:24:33.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f15700a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:33.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f157013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:33.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f15701430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:33.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157006ffa0 con 0x7f15700a98f0 2024-03-25T14:24:33.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157006e710 con 0x7f15701430c0 2024-03-25T14:24:33.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157014e6c0 con 0x7f157013f710 2024-03-25T14:24:33.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:33.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.072+0000 7f1577e0f640 10 monclient(hunting): authenticate will time out at 1636.217469s 2024-03-25T14:24:33.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.085+0000 7f15763ac640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f15701430c0 0x7f15701e60c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:58118/0 (socket says 172.21.15.63:58118) 2024-03-25T14:24:33.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.085+0000 7f15763ac640 1 -- 172.21.15.63:0/143980042 learned_addr learned my addr 172.21.15.63:0/143980042 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:33.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.090+0000 7f15667fc640 10 client.?.objecter ms_handle_connect 0x7f15701430c0 2024-03-25T14:24:33.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.090+0000 7f15667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:33.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.094+0000 7f15667fc640 10 client.?.objecter ms_handle_connect 0x7f15700a98f0 2024-03-25T14:24:33.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.094+0000 7f15667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:33.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.098+0000 7f15667fc640 10 client.?.objecter ms_handle_connect 0x7f157013f710 2024-03-25T14:24:33.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.098+0000 7f15667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:33.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.101+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679702680 0 0) 0x7f157006e710 con 0x7f15701430c0 2024-03-25T14:24:33.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.101+0000 7f15667fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:33.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.101+0000 7f15667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:33.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.101+0000 7f15667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:33.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.101+0000 7f15667fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T14:24:33.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.101+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1554004eb0 con 0x7f15701430c0 2024-03-25T14:24:33.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.103+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54260395 0 0) 0x7f157014e6c0 con 0x7f157013f710 2024-03-25T14:24:33.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.103+0000 7f15667fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:33.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.103+0000 7f15667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:33.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.103+0000 7f15667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:33.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.103+0000 7f15667fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T14:24:33.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.103+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157006e710 con 0x7f157013f710 2024-03-25T14:24:33.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.112+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142631757 0 0) 0x7f1554004eb0 con 0x7f15701430c0 2024-03-25T14:24:33.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.112+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f157014e6c0 con 0x7f15701430c0 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.113+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1568001990 con 0x7f15701430c0 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.113+0000 7f15667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.113+0000 7f15667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.113+0000 7f15667fc640 10 monclient(hunting): dump: 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:33.117 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:33.118 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:33.118 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:33.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.113+0000 7f15667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:33.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.117+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956189315 0 0) 0x7f157006ffa0 con 0x7f15700a98f0 2024-03-25T14:24:33.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.117+0000 7f15667fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:33.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.117+0000 7f15667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:33.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.117+0000 7f15667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:33.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.117+0000 7f15667fc640 10 monclient(hunting): my global_id is 4434 2024-03-25T14:24:33.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.117+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1554004eb0 con 0x7f15700a98f0 2024-03-25T14:24:33.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1791491125 0 0) 0x7f157014e6c0 con 0x7f15701430c0 2024-03-25T14:24:33.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 >> v1:172.21.15.63:6790/0 conn(0x7f157013f710 legacy=0x7f15700706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:33.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 >> v1:172.21.15.63:6789/0 conn(0x7f15700a98f0 legacy=0x7f157006ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:33.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 10 monclient: _finish_hunting 0 2024-03-25T14:24:33.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 1 monclient: found mon.b 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570139940 con 0x7f15701430c0 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 10 monclient: _finish_auth 0 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 10 monclient: _check_auth_tickets 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f15667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 5 monclient: authenticate success, global_id 4438 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 10 monclient: _renew_subs 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157014d2c0 con 0x7f15701430c0 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 10 monclient: _renew_subs 2024-03-25T14:24:33.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:33.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f157006e2f0 con 0x7f15701430c0 2024-03-25T14:24:33.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.120+0000 7f1577e0f640 1 librados: init done 2024-03-25T14:24:33.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.121+0000 7f1577e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:33.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.121+0000 7f1577e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:33.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.121+0000 7f1577e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:33.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.121+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157006e8a0 con 0x7f15701430c0 2024-03-25T14:24:33.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.341+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15680035f0 con 0x7f15701430c0 2024-03-25T14:24:33.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.341+0000 7f15667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:33.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.341+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f15680039c0 con 0x7f15701430c0 2024-03-25T14:24:33.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.341+0000 7f15667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:33.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.341+0000 7f15667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:33.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.341+0000 7f15667fc640 10 monclient: dump: 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:33.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.341+0000 7f1567fff640 4 set_mon_vals no callback set 2024-03-25T14:24:33.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.402+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f1568003b20 con 0x7f15701430c0 2024-03-25T14:24:33.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.402+0000 7f1575bab640 10 monclient: get_auth_request con 0x7f155406de00 auth_method 0 2024-03-25T14:24:33.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.403+0000 7f15667fc640 10 client.4438.objecter ms_handle_connect 0x7f155406de00 2024-03-25T14:24:33.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.406+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f1568054790 con 0x7f15701430c0 2024-03-25T14:24:33.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.406+0000 7f15667fc640 10 client.4438.objecter ms_dispatch 0x7f1570000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:33.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.406+0000 7f15667fc640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:33.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.406+0000 7f15667fc640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:33.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.406+0000 7f15667fc640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T14:24:33.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.410+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15680502a0 con 0x7f15701430c0 2024-03-25T14:24:33.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.410+0000 7f15667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:33.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.410+0000 7f15667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:33.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.645+0000 7f1577e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:33.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.645+0000 7f1577e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:33.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.645+0000 7f1577e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:33.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.645+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f157014f2d0 con 0x7f15701430c0 2024-03-25T14:24:33.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.768+0000 7f15667fc640 1 -- 172.21.15.63:0/143980042 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f15680521d0 con 0x7f15701430c0 2024-03-25T14:24:33.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.768+0000 7f15667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:33.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.768+0000 7f15667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:33.773 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:33.773 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:33.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 10 librados: watch_flush enter 2024-03-25T14:24:33.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 10 librados: watch_flush exit 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 10 client.4438.objecter successfully canceled tick 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 >> v1:172.21.15.153:6800/1688255552 conn(0x7f155406de00 legacy=0x7f15540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 10 monclient: shutdown 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 >> v1:172.21.15.153:6789/0 conn(0x7f15701430c0 legacy=0x7f15701e60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 shutdown_connections 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 >> 172.21.15.63:0/143980042 conn(0x7f15700b7ea0 msgr2=0x7f1570074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 shutdown_connections 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 1 -- 172.21.15.63:0/143980042 wait complete. 2024-03-25T14:24:33.776 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:33.771+0000 7f1577e0f640 1 librados: shutdown 2024-03-25T14:24:33.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:40.593 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:40.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.702+0000 7fca68893640 10 monclient: get_monmap_and_config 2024-03-25T14:24:40.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.702+0000 7fca68893640 10 monclient: build_initial_monmap 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.702+0000 7fca68893640 10 monclient: monmap: 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:40.703529+0000 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:40.703529+0000 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:40.707 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:40.708 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:40.708 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:40.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.703+0000 7fca68893640 1 Processor -- start 2024-03-25T14:24:40.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.703+0000 7fca68893640 1 -- start start 2024-03-25T14:24:40.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.703+0000 7fca68893640 10 monclient: init 2024-03-25T14:24:40.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.703+0000 7fca68893640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:40.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.703+0000 7fca68893640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:24:40.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 10 monclient(hunting): picked mon.noname-c con 0x7fca640a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:40.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 10 monclient(hunting): picked mon.noname-a con 0x7fca6413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:40.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 10 monclient(hunting): picked mon.noname-b con 0x7fca641430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:40.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca64142520 con 0x7fca6413f710 2024-03-25T14:24:40.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b28a0 con 0x7fca640a98f0 2024-03-25T14:24:40.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b2a30 con 0x7fca641430c0 2024-03-25T14:24:40.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:40.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.704+0000 7fca68893640 10 monclient(hunting): authenticate will time out at 1643.848768s 2024-03-25T14:24:40.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.710+0000 7fca62d9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fca641430c0 0x7fca6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:50252/0 (socket says 172.21.15.63:50252) 2024-03-25T14:24:40.714 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.710+0000 7fca62d9d640 1 -- 172.21.15.63:0/1650634155 learned_addr learned my addr 172.21.15.63:0/1650634155 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:40.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.718+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525002931 0 0) 0x7fca640b2a30 con 0x7fca641430c0 2024-03-25T14:24:40.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.718+0000 7fca61d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:24:40.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.718+0000 7fca61d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:40.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.718+0000 7fca61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:40.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.718+0000 7fca61d9b640 10 monclient(hunting): my global_id is 4421 2024-03-25T14:24:40.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.718+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca50003520 con 0x7fca641430c0 2024-03-25T14:24:40.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.727+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123191530 0 0) 0x7fca64142520 con 0x7fca6413f710 2024-03-25T14:24:40.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.727+0000 7fca61d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:24:40.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.727+0000 7fca61d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:40.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.727+0000 7fca61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:40.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.727+0000 7fca61d9b640 10 monclient(hunting): my global_id is 4437 2024-03-25T14:24:40.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.727+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca640b2a30 con 0x7fca6413f710 2024-03-25T14:24:40.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 793048733 0 0) 0x7fca50003520 con 0x7fca641430c0 2024-03-25T14:24:40.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 >> v1:172.21.15.153:6789/0 conn(0x7fca640a98f0 legacy=0x7fca640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:40.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 >> v1:172.21.15.63:6789/0 conn(0x7fca6413f710 legacy=0x7fca641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:40.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 10 monclient: _finish_hunting 0 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 1 monclient: found mon.noname-b 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640b44c0 con 0x7fca641430c0 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 10 monclient: _finish_auth 0 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 10 monclient: _check_auth_tickets 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca5c001a10 con 0x7fca641430c0 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.730+0000 7fca61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.731+0000 7fca61d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.731+0000 7fca61d9b640 10 monclient: dump: 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:40.736 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.731+0000 7fca68893640 5 monclient: authenticate success, global_id 4421 2024-03-25T14:24:40.737 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.731+0000 7fca68893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:40.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca5c001b70 con 0x7fca641430c0 2024-03-25T14:24:40.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca61d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:40.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca61d9b640 1 -- 172.21.15.63:0/1650634155 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca5c001f80 con 0x7fca641430c0 2024-03-25T14:24:40.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:40.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca61d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca61d9b640 10 monclient: dump: 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca68893640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:40.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.741+0000 7fca68893640 4 set_mon_vals no callback set 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.742+0000 7fca68893640 10 monclient: shutdown 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.742+0000 7fca68893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.742+0000 7fca68893640 1 -- 172.21.15.63:0/1650634155 >> v1:172.21.15.63:6790/0 conn(0x7fca641430c0 legacy=0x7fca6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.742+0000 7fca68893640 1 -- 172.21.15.63:0/1650634155 shutdown_connections 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.742+0000 7fca68893640 1 -- 172.21.15.63:0/1650634155 >> 172.21.15.63:0/1650634155 conn(0x7fca640b7ea0 msgr2=0x7fca640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.742+0000 7fca68893640 1 -- 172.21.15.63:0/1650634155 shutdown_connections 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.742+0000 7fca68893640 1 -- 172.21.15.63:0/1650634155 wait complete. 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.743+0000 7fca68893640 10 monclient: build_initial_monmap 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.743+0000 7fca68893640 10 monclient: monmap: 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:40.749 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 1 librados: starting msgr at 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 1 librados: starting objecter 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 1 Processor -- start 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 1 -- start start 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 1 librados: setting wanted keys 2024-03-25T14:24:40.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 1 librados: calling monclient init 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 10 monclient: init 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.744+0000 7fca68893640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 10 monclient(hunting): picked mon.noname-c con 0x7fca640a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 10 monclient(hunting): picked mon.noname-a con 0x7fca6413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 10 monclient(hunting): picked mon.noname-b con 0x7fca641430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6413c9b0 con 0x7fca6413f710 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca641397c0 con 0x7fca641430c0 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca64139950 con 0x7fca640a98f0 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:40.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.745+0000 7fca68893640 10 monclient(hunting): authenticate will time out at 1643.889794s 2024-03-25T14:24:40.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.749+0000 7fca5bfff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fca6413f710 0x7fca641dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:47292/0 (socket says 172.21.15.63:47292) 2024-03-25T14:24:40.753 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.749+0000 7fca5bfff640 1 -- 172.21.15.63:0/1602048085 learned_addr learned my addr 172.21.15.63:0/1602048085 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:40.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.753+0000 7fca5affd640 10 client.?.objecter ms_handle_connect 0x7fca640a98f0 2024-03-25T14:24:40.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.753+0000 7fca5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:40.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.755+0000 7fca5affd640 10 client.?.objecter ms_handle_connect 0x7fca6413f710 2024-03-25T14:24:40.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.755+0000 7fca5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:40.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.760+0000 7fca5affd640 10 client.?.objecter ms_handle_connect 0x7fca641430c0 2024-03-25T14:24:40.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.760+0000 7fca5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:40.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.765+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050405608 0 0) 0x7fca64139950 con 0x7fca640a98f0 2024-03-25T14:24:40.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.765+0000 7fca5affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:40.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.765+0000 7fca5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:40.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.765+0000 7fca5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:40.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.765+0000 7fca5affd640 10 monclient(hunting): my global_id is 4424 2024-03-25T14:24:40.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.765+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca44004eb0 con 0x7fca640a98f0 2024-03-25T14:24:40.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.768+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245948854 0 0) 0x7fca6413c9b0 con 0x7fca6413f710 2024-03-25T14:24:40.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.768+0000 7fca5affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:40.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.768+0000 7fca5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:40.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.768+0000 7fca5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:40.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.768+0000 7fca5affd640 10 monclient(hunting): my global_id is 4440 2024-03-25T14:24:40.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.768+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64139950 con 0x7fca6413f710 2024-03-25T14:24:40.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.776+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944634268 0 0) 0x7fca641397c0 con 0x7fca641430c0 2024-03-25T14:24:40.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.776+0000 7fca5affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:40.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.776+0000 7fca5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:40.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.776+0000 7fca5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:40.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.776+0000 7fca5affd640 10 monclient(hunting): my global_id is 4444 2024-03-25T14:24:40.780 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.776+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca6413c9b0 con 0x7fca641430c0 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.781+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485841132 0 0) 0x7fca44004eb0 con 0x7fca640a98f0 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.781+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca641397c0 con 0x7fca640a98f0 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.781+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca540018c0 con 0x7fca640a98f0 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.781+0000 7fca5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.781+0000 7fca5affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.781+0000 7fca5affd640 10 monclient(hunting): dump: 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:40.785 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:40.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.781+0000 7fca5affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.791+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4158362953 0 0) 0x7fca64139950 con 0x7fca6413f710 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.791+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca44004eb0 con 0x7fca6413f710 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.791+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca4c0019f0 con 0x7fca6413f710 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.791+0000 7fca5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.791+0000 7fca5affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.791+0000 7fca5affd640 10 monclient(hunting): dump: 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:40.795 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:40.796 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:40.796 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:40.796 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:40.796 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:40.796 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:40.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2544231624 0 0) 0x7fca641397c0 con 0x7fca640a98f0 2024-03-25T14:24:40.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 >> v1:172.21.15.153:6789/0 conn(0x7fca641430c0 legacy=0x7fca641e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 >> v1:172.21.15.63:6789/0 conn(0x7fca6413f710 legacy=0x7fca641dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 10 monclient: _finish_hunting 0 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 1 monclient: found mon.c 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca64131680 con 0x7fca640a98f0 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 10 monclient: _finish_auth 0 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 10 monclient: _check_auth_tickets 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 5 monclient: authenticate success, global_id 4424 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 10 monclient: _renew_subs 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca64131950 con 0x7fca640a98f0 2024-03-25T14:24:40.798 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 10 monclient: _renew_subs 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca64135a30 con 0x7fca640a98f0 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.792+0000 7fca68893640 1 librados: init done 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.793+0000 7fca68893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.793+0000 7fca68893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.793+0000 7fca68893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:40.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:40.793+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6413c810 con 0x7fca640a98f0 2024-03-25T14:24:41.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.113+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca540035f0 con 0x7fca640a98f0 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.113+0000 7fca5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.113+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca54003760 con 0x7fca640a98f0 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.113+0000 7fca5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.114+0000 7fca5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.114+0000 7fca5affd640 10 monclient: dump: 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:41.118 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:41.119 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:41.119 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:41.119 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:41.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.113+0000 7fca6159a640 4 set_mon_vals no callback set 2024-03-25T14:24:41.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.114+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fca54019730 con 0x7fca640a98f0 2024-03-25T14:24:41.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.115+0000 7fca5bfff640 10 monclient: get_auth_request con 0x7fca4406ded0 auth_method 0 2024-03-25T14:24:41.120 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.115+0000 7fca5affd640 10 client.4424.objecter ms_handle_connect 0x7fca4406ded0 2024-03-25T14:24:41.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fca54053480 con 0x7fca640a98f0 2024-03-25T14:24:41.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 10 client.4424.objecter ms_dispatch 0x7fca64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:41.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:41.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:41.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T14:24:41.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca5404edc0 con 0x7fca640a98f0 2024-03-25T14:24:41.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:41.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.121+0000 7fca5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:41.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.344+0000 7fca68893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:41.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.344+0000 7fca68893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:41.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.344+0000 7fca68893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:41.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.344+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca64136fc0 con 0x7fca640a98f0 2024-03-25T14:24:41.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.422+0000 7fca5affd640 1 -- 172.21.15.63:0/1602048085 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fca5404edc0 con 0x7fca640a98f0 2024-03-25T14:24:41.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.422+0000 7fca5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:41.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.422+0000 7fca5affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:41.426 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:41.426 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 10 librados: watch_flush enter 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 10 librados: watch_flush exit 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 10 client.4424.objecter successfully canceled tick 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 >> v1:172.21.15.153:6800/1688255552 conn(0x7fca4406ded0 legacy=0x7fca44070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 10 monclient: shutdown 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 >> v1:172.21.15.63:6790/0 conn(0x7fca640a98f0 legacy=0x7fca64130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 shutdown_connections 2024-03-25T14:24:41.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 >> 172.21.15.63:0/1602048085 conn(0x7fca640b7ea0 msgr2=0x7fca6414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:41.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.424+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 shutdown_connections 2024-03-25T14:24:41.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.425+0000 7fca68893640 1 -- 172.21.15.63:0/1602048085 wait complete. 2024-03-25T14:24:41.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:41.425+0000 7fca68893640 1 librados: shutdown 2024-03-25T14:24:41.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:46.690 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:25.202 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.733 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:25.193 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.734 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:25.247 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.734 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:25.194 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.758 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:25.217 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.770 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:25.283 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.796 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:25.309 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.807 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:25.320 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:46.838 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:25.351 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:24:48.247 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.359+0000 7f6c3c13d640 10 monclient: get_monmap_and_config 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.359+0000 7f6c3c13d640 10 monclient: build_initial_monmap 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.359+0000 7f6c3c13d640 10 monclient: monmap: 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:48.359410+0000 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:48.359410+0000 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:48.362 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:48.363 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:48.363 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:48.363 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:48.363 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.359+0000 7f6c3c13d640 1 Processor -- start 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.360+0000 7f6c3c13d640 1 -- start start 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.360+0000 7f6c3c13d640 10 monclient: init 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 10 monclient(hunting): picked mon.noname-c con 0x7f6c340a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 10 monclient(hunting): picked mon.noname-b con 0x7f6c3413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:48.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 10 monclient(hunting): picked mon.noname-a con 0x7f6c341430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:48.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34142520 con 0x7f6c341430c0 2024-03-25T14:24:48.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340b28a0 con 0x7f6c340a98f0 2024-03-25T14:24:48.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340b2a30 con 0x7f6c3413f710 2024-03-25T14:24:48.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:48.365 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.361+0000 7f6c3c13d640 10 monclient(hunting): authenticate will time out at 1651.504786s 2024-03-25T14:24:48.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.377+0000 7f6c396d8640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f6c3413f710 0x7f6c341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:54202/0 (socket says 172.21.15.63:54202) 2024-03-25T14:24:48.380 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.377+0000 7f6c396d8640 1 -- 172.21.15.63:0/3533830159 learned_addr learned my addr 172.21.15.63:0/3533830159 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:48.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.388+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976290653 0 0) 0x7f6c340b2a30 con 0x7f6c3413f710 2024-03-25T14:24:48.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.388+0000 7f6c38ed7640 10 monclient(hunting): _negotiate 2024-03-25T14:24:48.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.388+0000 7f6c38ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:48.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.388+0000 7f6c38ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:48.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.388+0000 7f6c38ed7640 10 monclient(hunting): my global_id is 4427 2024-03-25T14:24:48.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.388+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c20003520 con 0x7f6c3413f710 2024-03-25T14:24:48.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.395+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201569011 0 0) 0x7f6c340b28a0 con 0x7f6c340a98f0 2024-03-25T14:24:48.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.395+0000 7f6c38ed7640 10 monclient(hunting): _negotiate 2024-03-25T14:24:48.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.395+0000 7f6c38ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:48.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.395+0000 7f6c38ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:48.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.395+0000 7f6c38ed7640 10 monclient(hunting): my global_id is 4447 2024-03-25T14:24:48.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.395+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c340b2a30 con 0x7f6c340a98f0 2024-03-25T14:24:48.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.403+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213230803 0 0) 0x7f6c34142520 con 0x7f6c341430c0 2024-03-25T14:24:48.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.403+0000 7f6c38ed7640 10 monclient(hunting): _negotiate 2024-03-25T14:24:48.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.403+0000 7f6c38ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:48.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.403+0000 7f6c38ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:48.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.403+0000 7f6c38ed7640 10 monclient(hunting): my global_id is 4443 2024-03-25T14:24:48.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.403+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c340b28a0 con 0x7f6c341430c0 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701331871 0 0) 0x7f6c340b2a30 con 0x7f6c340a98f0 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 >> v1:172.21.15.63:6790/0 conn(0x7f6c3413f710 legacy=0x7f6c341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 >> v1:172.21.15.63:6789/0 conn(0x7f6c341430c0 legacy=0x7f6c3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 10 monclient: _finish_hunting 0 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 1 monclient: found mon.noname-c 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340b44c0 con 0x7f6c340a98f0 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 10 monclient: _finish_auth 0 2024-03-25T14:24:48.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 10 monclient: _check_auth_tickets 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6c30001670 con 0x7f6c340a98f0 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c38ed7640 10 monclient: dump: 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:48.411 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:48.412 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c3c13d640 5 monclient: authenticate success, global_id 4447 2024-03-25T14:24:48.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.407+0000 7f6c3c13d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:48.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c300017d0 con 0x7f6c340a98f0 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c38ed7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c38ed7640 1 -- 172.21.15.63:0/3533830159 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6c30001be0 con 0x7f6c340a98f0 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c3c13d640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c3c13d640 4 set_mon_vals no callback set 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c38ed7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c38ed7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c38ed7640 10 monclient: dump: 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.427 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:48.428 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:48.428 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:48.428 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:48.428 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:48.428 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c3c13d640 10 monclient: shutdown 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c3c13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c3c13d640 1 -- 172.21.15.63:0/3533830159 >> v1:172.21.15.153:6789/0 conn(0x7f6c340a98f0 legacy=0x7f6c340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c3c13d640 1 -- 172.21.15.63:0/3533830159 shutdown_connections 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.424+0000 7f6c3c13d640 1 -- 172.21.15.63:0/3533830159 >> 172.21.15.63:0/3533830159 conn(0x7f6c340b7ea0 msgr2=0x7f6c340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.425+0000 7f6c3c13d640 1 -- 172.21.15.63:0/3533830159 shutdown_connections 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.425+0000 7f6c3c13d640 1 -- 172.21.15.63:0/3533830159 wait complete. 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.425+0000 7f6c3c13d640 10 monclient: build_initial_monmap 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.425+0000 7f6c3c13d640 10 monclient: monmap: 2024-03-25T14:24:48.430 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 librados: starting msgr at 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 librados: starting objecter 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 Processor -- start 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 -- start start 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 librados: setting wanted keys 2024-03-25T14:24:48.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 librados: calling monclient init 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 10 monclient: init 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 10 monclient(hunting): picked mon.noname-b con 0x7f6c340a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 10 monclient(hunting): picked mon.noname-c con 0x7f6c341430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3413a3e0 con 0x7f6c3413f710 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34139ec0 con 0x7f6c340a98f0 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.426+0000 7f6c3c13d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34129930 con 0x7f6c341430c0 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.427+0000 7f6c3c13d640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:48.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.427+0000 7f6c3c13d640 10 monclient(hunting): authenticate will time out at 1651.570527s 2024-03-25T14:24:48.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.438+0000 7f6c396d8640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f6c3413f710 0x7f6c34128b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51342/0 (socket says 172.21.15.63:51342) 2024-03-25T14:24:48.441 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.438+0000 7f6c396d8640 1 -- 172.21.15.63:0/1303317559 learned_addr learned my addr 172.21.15.63:0/1303317559 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:48.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.444+0000 7f6c2a7fc640 10 client.?.objecter ms_handle_connect 0x7f6c3413f710 2024-03-25T14:24:48.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.444+0000 7f6c2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:48.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.449+0000 7f6c2a7fc640 10 client.?.objecter ms_handle_connect 0x7f6c341430c0 2024-03-25T14:24:48.452 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.449+0000 7f6c2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:48.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.455+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165909448 0 0) 0x7f6c3413a3e0 con 0x7f6c3413f710 2024-03-25T14:24:48.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.455+0000 7f6c2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:48.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.455+0000 7f6c2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:48.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.455+0000 7f6c2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:48.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.455+0000 7f6c2a7fc640 10 monclient(hunting): my global_id is 4446 2024-03-25T14:24:48.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.455+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c18004eb0 con 0x7f6c3413f710 2024-03-25T14:24:48.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.459+0000 7f6c2a7fc640 10 client.?.objecter ms_handle_connect 0x7f6c340a98f0 2024-03-25T14:24:48.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.459+0000 7f6c2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954602220 0 0) 0x7f6c34129930 con 0x7f6c341430c0 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3413a3e0 con 0x7f6c341430c0 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111052694 0 0) 0x7f6c34139ec0 con 0x7f6c340a98f0 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:24:48.471 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:48.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:48.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T14:24:48.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.468+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c34129930 con 0x7f6c340a98f0 2024-03-25T14:24:48.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.486+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847248485 0 0) 0x7f6c34129930 con 0x7f6c340a98f0 2024-03-25T14:24:48.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.486+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c34139ec0 con 0x7f6c340a98f0 2024-03-25T14:24:48.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.486+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6c30001ae0 con 0x7f6c340a98f0 2024-03-25T14:24:48.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.486+0000 7f6c2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:48.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.486+0000 7f6c2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.486+0000 7f6c2a7fc640 10 monclient(hunting): dump: 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.486+0000 7f6c2a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.490+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 609713492 0 0) 0x7f6c3413a3e0 con 0x7f6c341430c0 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.490+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c34129930 con 0x7f6c341430c0 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.490+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6c2c001b50 con 0x7f6c341430c0 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.490+0000 7f6c2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.490+0000 7f6c2a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.490+0000 7f6c2a7fc640 10 monclient(hunting): dump: 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:48.493 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:48.494 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:48.494 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:48.494 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.492+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2527991409 0 0) 0x7f6c18004eb0 con 0x7f6c3413f710 2024-03-25T14:24:48.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.492+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c3413a3e0 con 0x7f6c3413f710 2024-03-25T14:24:48.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.492+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6c24001950 con 0x7f6c3413f710 2024-03-25T14:24:48.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.492+0000 7f6c2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:48.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.492+0000 7f6c2a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:24:48.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.492+0000 7f6c2a7fc640 10 monclient(hunting): dump: 2024-03-25T14:24:48.495 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:48.496 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3264532032 0 0) 0x7f6c34139ec0 con 0x7f6c340a98f0 2024-03-25T14:24:48.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 >> v1:172.21.15.63:6790/0 conn(0x7f6c341430c0 legacy=0x7f6c34129220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:48.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 >> v1:172.21.15.63:6789/0 conn(0x7f6c3413f710 legacy=0x7f6c34128b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 1 monclient: found mon.b 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c34129ac0 con 0x7f6c340a98f0 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 5 monclient: authenticate success, global_id 4450 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 10 monclient: _renew_subs 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c340b79f0 con 0x7f6c340a98f0 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 10 monclient: _renew_subs 2024-03-25T14:24:48.505 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:48.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3412b2f0 con 0x7f6c340a98f0 2024-03-25T14:24:48.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.501+0000 7f6c3c13d640 1 librados: init done 2024-03-25T14:24:48.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.502+0000 7f6c3c13d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:48.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.502+0000 7f6c3c13d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:48.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.502+0000 7f6c3c13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:48.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.502+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c340b3ef0 con 0x7f6c340a98f0 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c30001210 con 0x7f6c340a98f0 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6c30003a40 con 0x7f6c340a98f0 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2a7fc640 10 monclient: dump: 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:48.800 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:48.801 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:48.801 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:48.801 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:48.801 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:48.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2bfff640 4 set_mon_vals no callback set 2024-03-25T14:24:48.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.797+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f6c30019a10 con 0x7f6c340a98f0 2024-03-25T14:24:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.798+0000 7f6c396d8640 10 monclient: get_auth_request con 0x7f6c1806e0b0 auth_method 0 2024-03-25T14:24:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.798+0000 7f6c2a7fc640 10 client.4450.objecter ms_handle_connect 0x7f6c1806e0b0 2024-03-25T14:24:48.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f6c30053650 con 0x7f6c340a98f0 2024-03-25T14:24:48.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 10 client.4450.objecter ms_dispatch 0x7f6c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:48.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:48.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:48.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T14:24:48.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c3004ef90 con 0x7f6c340a98f0 2024-03-25T14:24:48.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:48.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:48.803+0000 7f6c2a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:49.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.060+0000 7f6c3c13d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:49.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.060+0000 7f6c3c13d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:49.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.060+0000 7f6c3c13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:24:49.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.060+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c34139d20 con 0x7f6c340a98f0 2024-03-25T14:24:49.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.096+0000 7f6c2a7fc640 1 -- 172.21.15.63:0/1303317559 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f6c34139d20 con 0x7f6c340a98f0 2024-03-25T14:24:49.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.096+0000 7f6c2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:49.099 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.096+0000 7f6c2a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:49.100 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:49.100 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:49.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 10 librados: watch_flush enter 2024-03-25T14:24:49.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 10 librados: watch_flush exit 2024-03-25T14:24:49.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T14:24:49.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 10 client.4450.objecter successfully canceled tick 2024-03-25T14:24:49.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 >> v1:172.21.15.153:6800/1688255552 conn(0x7f6c1806e0b0 legacy=0x7f6c18070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 10 monclient: shutdown 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 >> v1:172.21.15.153:6789/0 conn(0x7f6c340a98f0 legacy=0x7f6c3414d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 shutdown_connections 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 >> 172.21.15.63:0/1303317559 conn(0x7f6c340b7ea0 msgr2=0x7f6c34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 shutdown_connections 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 1 -- 172.21.15.63:0/1303317559 wait complete. 2024-03-25T14:24:49.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:49.099+0000 7f6c3c13d640 1 librados: shutdown 2024-03-25T14:24:49.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:24:55.923 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:24:56.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.029+0000 7f1e9c4d3640 10 monclient: get_monmap_and_config 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.029+0000 7f1e9c4d3640 10 monclient: build_initial_monmap 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.029+0000 7f1e9c4d3640 10 monclient: monmap: 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:24:56.031025+0000 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:24:56.031025+0000 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:24:56.034 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.030+0000 7f1e9c4d3640 1 Processor -- start 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.030+0000 7f1e9c4d3640 1 -- start start 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.030+0000 7f1e9c4d3640 10 monclient: init 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f1e940a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f1e9413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:56.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f1e941430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:56.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e94142520 con 0x7f1e9413f710 2024-03-25T14:24:56.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e940b28a0 con 0x7f1e940a98f0 2024-03-25T14:24:56.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e940b2a30 con 0x7f1e941430c0 2024-03-25T14:24:56.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:56.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.031+0000 7f1e9c4d3640 10 monclient(hunting): authenticate will time out at 1659.176086s 2024-03-25T14:24:56.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.041+0000 7f1e9a26f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1e940a98f0 0x7f1e940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:50900/0 (socket says 172.21.15.63:50900) 2024-03-25T14:24:56.044 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.041+0000 7f1e9a26f640 1 -- 172.21.15.63:0/1357852493 learned_addr learned my addr 172.21.15.63:0/1357852493 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:56.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.049+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397727692 0 0) 0x7f1e94142520 con 0x7f1e9413f710 2024-03-25T14:24:56.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.049+0000 7f1e9926d640 10 monclient(hunting): _negotiate 2024-03-25T14:24:56.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.049+0000 7f1e9926d640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:56.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.049+0000 7f1e9926d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:56.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.049+0000 7f1e9926d640 10 monclient(hunting): my global_id is 4449 2024-03-25T14:24:56.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.049+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e80003520 con 0x7f1e9413f710 2024-03-25T14:24:56.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.052+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175892824 0 0) 0x7f1e940b2a30 con 0x7f1e941430c0 2024-03-25T14:24:56.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.052+0000 7f1e9926d640 10 monclient(hunting): _negotiate 2024-03-25T14:24:56.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.052+0000 7f1e9926d640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:56.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.052+0000 7f1e9926d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:56.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.052+0000 7f1e9926d640 10 monclient(hunting): my global_id is 4433 2024-03-25T14:24:56.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.052+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e94142520 con 0x7f1e941430c0 2024-03-25T14:24:56.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618358691 0 0) 0x7f1e80003520 con 0x7f1e9413f710 2024-03-25T14:24:56.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 >> v1:172.21.15.63:6790/0 conn(0x7f1e941430c0 legacy=0x7f1e9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:56.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 >> v1:172.21.15.153:6789/0 conn(0x7f1e940a98f0 legacy=0x7f1e940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 10 monclient: _finish_hunting 0 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 1 monclient: found mon.noname-a 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e940b44c0 con 0x7f1e9413f710 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 10 monclient: _finish_auth 0 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 10 monclient: _check_auth_tickets 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1e84001690 con 0x7f1e9413f710 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9c4d3640 5 monclient: authenticate success, global_id 4449 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9c4d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9926d640 10 monclient: dump: 2024-03-25T14:24:56.064 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:56.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.059+0000 7f1e9c4d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:24:56.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e840017f0 con 0x7f1e9413f710 2024-03-25T14:24:56.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9926d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:56.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9926d640 1 -- 172.21.15.63:0/1357852493 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1e84001c00 con 0x7f1e9413f710 2024-03-25T14:24:56.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9c4d3640 10 monclient: get_monmap_and_config success 2024-03-25T14:24:56.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9c4d3640 4 set_mon_vals no callback set 2024-03-25T14:24:56.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9926d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:56.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9926d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.068+0000 7f1e9926d640 10 monclient: dump: 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:56.073 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:56.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.069+0000 7f1e9c4d3640 10 monclient: shutdown 2024-03-25T14:24:56.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.069+0000 7f1e9c4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:56.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.069+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/1357852493 >> v1:172.21.15.63:6789/0 conn(0x7f1e9413f710 legacy=0x7f1e941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.069+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/1357852493 shutdown_connections 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.069+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/1357852493 >> 172.21.15.63:0/1357852493 conn(0x7f1e940b7ea0 msgr2=0x7f1e940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.070+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/1357852493 shutdown_connections 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.070+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/1357852493 wait complete. 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 10 monclient: build_initial_monmap 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 10 monclient: monmap: 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:24:56.075 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:24:56.076 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:24:56.076 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 1 librados: starting msgr at 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 1 librados: starting objecter 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 1 Processor -- start 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 1 -- start start 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 1 librados: setting wanted keys 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 1 librados: calling monclient init 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.071+0000 7f1e9c4d3640 10 monclient: init 2024-03-25T14:24:56.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 10 monclient: _reopen_session rank -1 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f1e940a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f1e9413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f1e941430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e94136e60 con 0x7f1e9413f710 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e94135a00 con 0x7f1e941430c0 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e941dc070 con 0x7f1e940a98f0 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 10 monclient(hunting): _renew_subs 2024-03-25T14:24:56.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.072+0000 7f1e9c4d3640 10 monclient(hunting): authenticate will time out at 1659.216938s 2024-03-25T14:24:56.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.077+0000 7f1e99a6e640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f1e9413f710 0x7f1e941db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59208/0 (socket says 172.21.15.63:59208) 2024-03-25T14:24:56.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.077+0000 7f1e99a6e640 1 -- 172.21.15.63:0/107579317 learned_addr learned my addr 172.21.15.63:0/107579317 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:24:56.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.083+0000 7f1e8affd640 10 client.?.objecter ms_handle_connect 0x7f1e9413f710 2024-03-25T14:24:56.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.083+0000 7f1e8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:56.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.085+0000 7f1e8affd640 10 client.?.objecter ms_handle_connect 0x7f1e940a98f0 2024-03-25T14:24:56.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.085+0000 7f1e8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:56.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.090+0000 7f1e8affd640 10 client.?.objecter ms_handle_connect 0x7f1e941430c0 2024-03-25T14:24:56.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.090+0000 7f1e8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:24:56.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.090+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459482084 0 0) 0x7f1e94136e60 con 0x7f1e9413f710 2024-03-25T14:24:56.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.091+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:56.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.091+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:56.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.091+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:56.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.091+0000 7f1e8affd640 10 monclient(hunting): my global_id is 4452 2024-03-25T14:24:56.095 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.091+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e78004eb0 con 0x7f1e9413f710 2024-03-25T14:24:56.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.098+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095999751 0 0) 0x7f1e941dc070 con 0x7f1e940a98f0 2024-03-25T14:24:56.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.098+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:56.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.098+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:56.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.098+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:56.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.098+0000 7f1e8affd640 10 monclient(hunting): my global_id is 4436 2024-03-25T14:24:56.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.098+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e94136e60 con 0x7f1e940a98f0 2024-03-25T14:24:56.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.099+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513215877 0 0) 0x7f1e94135a00 con 0x7f1e941430c0 2024-03-25T14:24:56.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.099+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-03-25T14:24:56.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.099+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:24:56.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.099+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:24:56.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.099+0000 7f1e8affd640 10 monclient(hunting): my global_id is 4456 2024-03-25T14:24:56.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.099+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e941dc070 con 0x7f1e941430c0 2024-03-25T14:24:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.107+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084129556 0 0) 0x7f1e94136e60 con 0x7f1e940a98f0 2024-03-25T14:24:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.107+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e94135a00 con 0x7f1e940a98f0 2024-03-25T14:24:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.107+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1e900018a0 con 0x7f1e940a98f0 2024-03-25T14:24:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.107+0000 7f1e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:56.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.107+0000 7f1e8affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.107+0000 7f1e8affd640 10 monclient(hunting): dump: 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:56.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.107+0000 7f1e8affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 821504860 0 0) 0x7f1e94135a00 con 0x7f1e940a98f0 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 >> v1:172.21.15.153:6789/0 conn(0x7f1e941430c0 legacy=0x7f1e941db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 >> v1:172.21.15.63:6789/0 conn(0x7f1e9413f710 legacy=0x7f1e941db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 10 monclient: _finish_hunting 0 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 1 monclient: found mon.c 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:56.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94137fa0 con 0x7f1e940a98f0 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 10 monclient: _finish_auth 0 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 10 monclient: _check_auth_tickets 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 5 monclient: authenticate success, global_id 4436 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 10 monclient: _renew_subs 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e941370b0 con 0x7f1e940a98f0 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 10 monclient: _renew_subs 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e940b6f80 con 0x7f1e940a98f0 2024-03-25T14:24:56.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.116+0000 7f1e9c4d3640 1 librados: init done 2024-03-25T14:24:56.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.118+0000 7f1e9c4d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:56.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.118+0000 7f1e9c4d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:56.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.118+0000 7f1e9c4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:56.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.118+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e940627b0 con 0x7f1e940a98f0 2024-03-25T14:24:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.442+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e90001cb0 con 0x7f1e940a98f0 2024-03-25T14:24:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.442+0000 7f1e8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:24:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.442+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1e90004980 con 0x7f1e940a98f0 2024-03-25T14:24:56.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.442+0000 7f1e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.442+0000 7f1e8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.442+0000 7f1e8affd640 10 monclient: dump: 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:24:56.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.442+0000 7f1e98a6c640 4 set_mon_vals no callback set 2024-03-25T14:24:56.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.443+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f1e90004b60 con 0x7f1e940a98f0 2024-03-25T14:24:56.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.443+0000 7f1e99a6e640 10 monclient: get_auth_request con 0x7f1e7806de20 auth_method 0 2024-03-25T14:24:56.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.443+0000 7f1e8affd640 10 client.4436.objecter ms_handle_connect 0x7f1e7806de20 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f1e900546a0 con 0x7f1e940a98f0 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 10 client.4436.objecter ms_dispatch 0x7f1e94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e9004ffe0 con 0x7f1e940a98f0 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:24:56.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.449+0000 7f1e8affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:24:56.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.694+0000 7f1e9c4d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:56.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.694+0000 7f1e9c4d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:56.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.694+0000 7f1e9c4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:24:56.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.694+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e9413e250 con 0x7f1e940a98f0 2024-03-25T14:24:56.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.715+0000 7f1e8affd640 1 -- 172.21.15.63:0/107579317 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f1e900520e0 con 0x7f1e940a98f0 2024-03-25T14:24:56.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.715+0000 7f1e8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:24:56.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.715+0000 7f1e8affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:24:56.720 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:24:56.720 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:24:56.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 10 librados: watch_flush enter 2024-03-25T14:24:56.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 10 librados: watch_flush exit 2024-03-25T14:24:56.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-25T14:24:56.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 10 client.4436.objecter successfully canceled tick 2024-03-25T14:24:56.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 >> v1:172.21.15.153:6800/1688255552 conn(0x7f1e7806de20 legacy=0x7f1e780702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:56.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 10 monclient: shutdown 2024-03-25T14:24:56.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:24:56.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.717+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 >> v1:172.21.15.63:6790/0 conn(0x7f1e940a98f0 legacy=0x7f1e94137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:24:56.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.718+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 shutdown_connections 2024-03-25T14:24:56.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.718+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 >> 172.21.15.63:0/107579317 conn(0x7f1e940b7ea0 msgr2=0x7f1e9414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:24:56.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.718+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 shutdown_connections 2024-03-25T14:24:56.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.718+0000 7f1e9c4d3640 1 -- 172.21.15.63:0/107579317 wait complete. 2024-03-25T14:24:56.723 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:24:56.718+0000 7f1e9c4d3640 1 librados: shutdown 2024-03-25T14:24:56.733 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:03.541 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.652+0000 7f5a50554640 10 monclient: get_monmap_and_config 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.652+0000 7f5a50554640 10 monclient: build_initial_monmap 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.652+0000 7f5a50554640 10 monclient: monmap: 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:03.654072+0000 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:03.654072+0000 2024-03-25T14:25:03.657 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:03.658 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:03.658 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:03.658 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:03.658 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:03.658 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.653+0000 7f5a50554640 1 Processor -- start 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.653+0000 7f5a50554640 1 -- start start 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.653+0000 7f5a50554640 10 monclient: init 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.654+0000 7f5a50554640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.654+0000 7f5a50554640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 10 monclient(hunting): picked mon.noname-b con 0x7f5a480a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 10 monclient(hunting): picked mon.noname-a con 0x7f5a481430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a48142520 con 0x7f5a481430c0 2024-03-25T14:25:03.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a480b28a0 con 0x7f5a4813f710 2024-03-25T14:25:03.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a480b2a30 con 0x7f5a480a98f0 2024-03-25T14:25:03.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:03.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.655+0000 7f5a50554640 10 monclient(hunting): authenticate will time out at 1666.799942s 2024-03-25T14:25:03.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.667+0000 7f5a4daef640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5a4813f710 0x7f5a481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:42436/0 (socket says 172.21.15.63:42436) 2024-03-25T14:25:03.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.667+0000 7f5a4daef640 1 -- 172.21.15.63:0/1587567106 learned_addr learned my addr 172.21.15.63:0/1587567106 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:03.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.682+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358583417 0 0) 0x7f5a480b28a0 con 0x7f5a4813f710 2024-03-25T14:25:03.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.682+0000 7f5a4d2ee640 10 monclient(hunting): _negotiate 2024-03-25T14:25:03.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.682+0000 7f5a4d2ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:03.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.682+0000 7f5a4d2ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:03.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.682+0000 7f5a4d2ee640 10 monclient(hunting): my global_id is 4459 2024-03-25T14:25:03.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.683+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a34003520 con 0x7f5a4813f710 2024-03-25T14:25:03.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.691+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211769958 0 0) 0x7f5a48142520 con 0x7f5a481430c0 2024-03-25T14:25:03.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.691+0000 7f5a4d2ee640 10 monclient(hunting): _negotiate 2024-03-25T14:25:03.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.691+0000 7f5a4d2ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:03.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.691+0000 7f5a4d2ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:03.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.691+0000 7f5a4d2ee640 10 monclient(hunting): my global_id is 4455 2024-03-25T14:25:03.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.691+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a480b28a0 con 0x7f5a481430c0 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.698+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 211845402 0 0) 0x7f5a34003520 con 0x7f5a4813f710 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 >> v1:172.21.15.63:6790/0 conn(0x7f5a480a98f0 legacy=0x7f5a480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 >> v1:172.21.15.63:6789/0 conn(0x7f5a481430c0 legacy=0x7f5a4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 10 monclient: _finish_hunting 0 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 1 monclient: found mon.noname-c 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a480b44c0 con 0x7f5a4813f710 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 10 monclient: _finish_auth 0 2024-03-25T14:25:03.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 10 monclient: _check_auth_tickets 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5a38001690 con 0x7f5a4813f710 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a50554640 5 monclient: authenticate success, global_id 4459 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a50554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a4d2ee640 10 monclient: dump: 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:03.704 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:03.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.699+0000 7f5a50554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.720+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a380017f0 con 0x7f5a4813f710 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.720+0000 7f5a4d2ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.720+0000 7f5a4d2ee640 1 -- 172.21.15.63:0/1587567106 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5a38001c00 con 0x7f5a4813f710 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.720+0000 7f5a50554640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a50554640 4 set_mon_vals no callback set 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a4d2ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a4d2ee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a4d2ee640 10 monclient: dump: 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.725 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.726 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:03.726 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:03.726 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:03.726 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:03.726 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:03.726 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:03.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a50554640 10 monclient: shutdown 2024-03-25T14:25:03.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a50554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:03.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a50554640 1 -- 172.21.15.63:0/1587567106 >> v1:172.21.15.153:6789/0 conn(0x7f5a4813f710 legacy=0x7f5a481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:03.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a50554640 1 -- 172.21.15.63:0/1587567106 shutdown_connections 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.721+0000 7f5a50554640 1 -- 172.21.15.63:0/1587567106 >> 172.21.15.63:0/1587567106 conn(0x7f5a480b7ea0 msgr2=0x7f5a480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.722+0000 7f5a50554640 1 -- 172.21.15.63:0/1587567106 shutdown_connections 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.722+0000 7f5a50554640 1 -- 172.21.15.63:0/1587567106 wait complete. 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.722+0000 7f5a50554640 10 monclient: build_initial_monmap 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.722+0000 7f5a50554640 10 monclient: monmap: 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:03.729 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.723+0000 7f5a50554640 1 librados: starting msgr at 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.723+0000 7f5a50554640 1 librados: starting objecter 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.723+0000 7f5a50554640 1 Processor -- start 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.723+0000 7f5a50554640 1 -- start start 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.723+0000 7f5a50554640 1 librados: setting wanted keys 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.723+0000 7f5a50554640 1 librados: calling monclient init 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.723+0000 7f5a50554640 10 monclient: init 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 10 monclient(hunting): picked mon.noname-a con 0x7f5a480a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4813f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 10 monclient(hunting): picked mon.noname-b con 0x7f5a481430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:03.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a48138a50 con 0x7f5a480a98f0 2024-03-25T14:25:03.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a480ab9e0 con 0x7f5a481430c0 2024-03-25T14:25:03.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a481d7880 con 0x7f5a4813f710 2024-03-25T14:25:03.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:03.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.724+0000 7f5a50554640 10 monclient(hunting): authenticate will time out at 1666.868764s 2024-03-25T14:25:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.726+0000 7f5a4eaf1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5a481430c0 0x7f5a481e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:42486/0 (socket says 172.21.15.63:42486) 2024-03-25T14:25:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.726+0000 7f5a4eaf1640 1 -- 172.21.15.63:0/2794491776 learned_addr learned my addr 172.21.15.63:0/2794491776 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.727+0000 7f5a3effd640 10 client.?.objecter ms_handle_connect 0x7f5a481430c0 2024-03-25T14:25:03.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.727+0000 7f5a3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:03.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.738+0000 7f5a3effd640 10 client.?.objecter ms_handle_connect 0x7f5a4813f710 2024-03-25T14:25:03.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.738+0000 7f5a3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:03.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.739+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146237425 0 0) 0x7f5a480ab9e0 con 0x7f5a481430c0 2024-03-25T14:25:03.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.739+0000 7f5a3effd640 10 monclient(hunting): _negotiate 2024-03-25T14:25:03.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.739+0000 7f5a3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:03.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.739+0000 7f5a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:03.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.739+0000 7f5a3effd640 10 monclient(hunting): my global_id is 4462 2024-03-25T14:25:03.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.739+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c004eb0 con 0x7f5a481430c0 2024-03-25T14:25:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 10 client.?.objecter ms_handle_connect 0x7f5a480a98f0 2024-03-25T14:25:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530075747 0 0) 0x7f5a481d7880 con 0x7f5a4813f710 2024-03-25T14:25:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 10 monclient(hunting): _negotiate 2024-03-25T14:25:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:03.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 10 monclient(hunting): my global_id is 4442 2024-03-25T14:25:03.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.751+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a480ab9e0 con 0x7f5a4813f710 2024-03-25T14:25:03.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.755+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080184407 0 0) 0x7f5a48138a50 con 0x7f5a480a98f0 2024-03-25T14:25:03.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.755+0000 7f5a3effd640 10 monclient(hunting): _negotiate 2024-03-25T14:25:03.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.755+0000 7f5a3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:03.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.755+0000 7f5a3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:03.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.755+0000 7f5a3effd640 10 monclient(hunting): my global_id is 4458 2024-03-25T14:25:03.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.755+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a481d7880 con 0x7f5a480a98f0 2024-03-25T14:25:03.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.760+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070702677 0 0) 0x7f5a2c004eb0 con 0x7f5a481430c0 2024-03-25T14:25:03.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.760+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a48138a50 con 0x7f5a481430c0 2024-03-25T14:25:03.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.760+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5a40001c90 con 0x7f5a481430c0 2024-03-25T14:25:03.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.760+0000 7f5a3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:03.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.760+0000 7f5a3effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.760+0000 7f5a3effd640 10 monclient(hunting): dump: 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:03.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.760+0000 7f5a3effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:03.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.765+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 906976955 0 0) 0x7f5a480ab9e0 con 0x7f5a4813f710 2024-03-25T14:25:03.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.765+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a2c004eb0 con 0x7f5a4813f710 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.766+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5a38001b40 con 0x7f5a4813f710 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.766+0000 7f5a3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.766+0000 7f5a3effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.766+0000 7f5a3effd640 10 monclient(hunting): dump: 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:03.770 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:03.771 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 662385918 0 0) 0x7f5a2c004eb0 con 0x7f5a4813f710 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 >> v1:172.21.15.153:6789/0 conn(0x7f5a481430c0 legacy=0x7f5a481e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 >> v1:172.21.15.63:6789/0 conn(0x7f5a480a98f0 legacy=0x7f5a48138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 10 monclient: _finish_hunting 0 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 1 monclient: found mon.c 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a481d84b0 con 0x7f5a4813f710 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 10 monclient: _finish_auth 0 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 10 monclient: _check_auth_tickets 2024-03-25T14:25:03.784 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 5 monclient: authenticate success, global_id 4442 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 10 monclient: _renew_subs 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a481d51f0 con 0x7f5a4813f710 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 10 monclient: _renew_subs 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a48138e60 con 0x7f5a4813f710 2024-03-25T14:25:03.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.779+0000 7f5a50554640 1 librados: init done 2024-03-25T14:25:03.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.781+0000 7f5a50554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:03.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.781+0000 7f5a50554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:03.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.781+0000 7f5a50554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:03.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:03.781+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a481d76e0 con 0x7f5a4813f710 2024-03-25T14:25:04.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.098+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a38004a90 con 0x7f5a4813f710 2024-03-25T14:25:04.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.098+0000 7f5a3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:04.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.098+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5a38001250 con 0x7f5a4813f710 2024-03-25T14:25:04.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.098+0000 7f5a3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:04.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.098+0000 7f5a3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:25:04.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.098+0000 7f5a3effd640 10 monclient: dump: 2024-03-25T14:25:04.102 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:04.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.098+0000 7f5a4caed640 4 set_mon_vals no callback set 2024-03-25T14:25:04.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.099+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f5a3801aa70 con 0x7f5a4813f710 2024-03-25T14:25:04.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.099+0000 7f5a4e2f0640 10 monclient: get_auth_request con 0x7f5a2c06df70 auth_method 0 2024-03-25T14:25:04.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.100+0000 7f5a3effd640 10 client.4442.objecter ms_handle_connect 0x7f5a2c06df70 2024-03-25T14:25:04.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.105+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f5a38054700 con 0x7f5a4813f710 2024-03-25T14:25:04.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.105+0000 7f5a3effd640 10 client.4442.objecter ms_dispatch 0x7f5a48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:04.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.105+0000 7f5a3effd640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:04.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.105+0000 7f5a3effd640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:04.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.105+0000 7f5a3effd640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T14:25:04.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.106+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a38050040 con 0x7f5a4813f710 2024-03-25T14:25:04.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.106+0000 7f5a3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:04.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.106+0000 7f5a3effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:04.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.343+0000 7f5a50554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:04.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.343+0000 7f5a50554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:04.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.343+0000 7f5a50554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:04.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.343+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a481d7a10 con 0x7f5a4813f710 2024-03-25T14:25:04.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.424+0000 7f5a3effd640 1 -- 172.21.15.63:0/2794491776 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f5a38052140 con 0x7f5a4813f710 2024-03-25T14:25:04.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.424+0000 7f5a3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:04.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.424+0000 7f5a3effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:04.429 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:04.429 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:04.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 10 librados: watch_flush enter 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 10 librados: watch_flush exit 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 10 client.4442.objecter successfully canceled tick 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 >> v1:172.21.15.153:6800/1688255552 conn(0x7f5a2c06df70 legacy=0x7f5a2c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 10 monclient: shutdown 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 >> v1:172.21.15.63:6790/0 conn(0x7f5a4813f710 legacy=0x7f5a481d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 shutdown_connections 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 >> 172.21.15.63:0/2794491776 conn(0x7f5a480b7ea0 msgr2=0x7f5a48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 shutdown_connections 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 1 -- 172.21.15.63:0/2794491776 wait complete. 2024-03-25T14:25:04.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:04.427+0000 7f5a50554640 1 librados: shutdown 2024-03-25T14:25:04.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:06.813 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:45.326 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.817 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:45.330 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.854 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:05:45.367 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.882 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:45.341 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.882 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:45.395 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.891 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:45.350 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.893 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:45.405 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.912 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:45.372 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:06.923 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:05:45.436 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:11.248 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.352+0000 7f0b5f59e640 10 monclient: get_monmap_and_config 2024-03-25T14:25:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.352+0000 7f0b5f59e640 10 monclient: build_initial_monmap 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.352+0000 7f0b5f59e640 10 monclient: monmap: 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:11.353942+0000 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:11.353942+0000 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:11.357 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.353+0000 7f0b5f59e640 1 Processor -- start 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.353+0000 7f0b5f59e640 1 -- start start 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.353+0000 7f0b5f59e640 10 monclient: init 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6006c2d0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6006c6b0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6014aeb0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600adee0 con 0x7f0b6006c6b0 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600ae1c0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6006d140 con 0x7f0b6014aeb0 2024-03-25T14:25:11.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:11.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5f59e640 10 monclient(hunting): authenticate will time out at 1674.498768s 2024-03-25T14:25:11.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5ed9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0b6014aeb0 0x7f0b6014d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:39158/0 (socket says 172.21.15.63:39158) 2024-03-25T14:25:11.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.354+0000 7f0b5ed9d640 1 -- 172.21.15.63:0/2236532483 learned_addr learned my addr 172.21.15.63:0/2236532483 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:11.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.413+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481135859 0 0) 0x7f0b600adee0 con 0x7f0b6006c6b0 2024-03-25T14:25:11.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.413+0000 7f0b5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:25:11.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.413+0000 7f0b5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:11.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.413+0000 7f0b5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:11.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.413+0000 7f0b5dd9b640 10 monclient(hunting): my global_id is 4461 2024-03-25T14:25:11.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.413+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c003520 con 0x7f0b6006c6b0 2024-03-25T14:25:11.429 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 500740011 0 0) 0x7f0b4c003520 con 0x7f0b6006c6b0 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 >> v1:172.21.15.63:6790/0 conn(0x7f0b6014aeb0 legacy=0x7f0b6014d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 >> v1:172.21.15.153:6789/0 conn(0x7f0b6006c2d0 legacy=0x7f0b600b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 10 monclient: _finish_hunting 0 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 1 monclient: found mon.noname-a 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6006d730 con 0x7f0b6006c6b0 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 10 monclient: _finish_auth 0 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 10 monclient: _check_auth_tickets 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0b48001690 con 0x7f0b6006c6b0 2024-03-25T14:25:11.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5dd9b640 10 monclient: dump: 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5f59e640 5 monclient: authenticate success, global_id 4461 2024-03-25T14:25:11.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.425+0000 7f0b5f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:11.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b480017f0 con 0x7f0b6006c6b0 2024-03-25T14:25:11.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:11.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5dd9b640 1 -- 172.21.15.63:0/2236532483 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0b48001c00 con 0x7f0b6006c6b0 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5f59e640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5f59e640 4 set_mon_vals no callback set 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.449+0000 7f0b5dd9b640 10 monclient: dump: 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:11.454 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:11.455 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:11.455 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.450+0000 7f0b5f59e640 10 monclient: shutdown 2024-03-25T14:25:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.450+0000 7f0b5f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.450+0000 7f0b5f59e640 1 -- 172.21.15.63:0/2236532483 >> v1:172.21.15.63:6789/0 conn(0x7f0b6006c6b0 legacy=0x7f0b600b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.450+0000 7f0b5f59e640 1 -- 172.21.15.63:0/2236532483 shutdown_connections 2024-03-25T14:25:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.450+0000 7f0b5f59e640 1 -- 172.21.15.63:0/2236532483 >> 172.21.15.63:0/2236532483 conn(0x7f0b600b5b90 msgr2=0x7f0b600b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.450+0000 7f0b5f59e640 1 -- 172.21.15.63:0/2236532483 shutdown_connections 2024-03-25T14:25:11.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.450+0000 7f0b5f59e640 1 -- 172.21.15.63:0/2236532483 wait complete. 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.451+0000 7f0b5f59e640 10 monclient: build_initial_monmap 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.451+0000 7f0b5f59e640 10 monclient: monmap: 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:11.458 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 1 librados: starting msgr at 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 1 librados: starting objecter 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 1 Processor -- start 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 1 -- start start 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 1 librados: setting wanted keys 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 1 librados: calling monclient init 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 10 monclient: init 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6006c2d0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.452+0000 7f0b5f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6006c6b0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b5f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6014aeb0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b5f59e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b60148000 con 0x7f0b6006c2d0 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b5f59e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b601db630 con 0x7f0b6006c6b0 2024-03-25T14:25:11.459 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b5f59e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b601db230 con 0x7f0b6014aeb0 2024-03-25T14:25:11.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b5f59e640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:11.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b5f59e640 10 monclient(hunting): authenticate will time out at 1674.597627s 2024-03-25T14:25:11.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b57fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0b6006c6b0 0x7f0b601486d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:46194/0 (socket says 172.21.15.63:46194) 2024-03-25T14:25:11.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.453+0000 7f0b57fff640 1 -- 172.21.15.63:0/4092378049 learned_addr learned my addr 172.21.15.63:0/4092378049 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:11.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.454+0000 7f0b56ffd640 10 client.?.objecter ms_handle_connect 0x7f0b6006c6b0 2024-03-25T14:25:11.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.454+0000 7f0b56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:11.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.462+0000 7f0b56ffd640 10 client.?.objecter ms_handle_connect 0x7f0b6014aeb0 2024-03-25T14:25:11.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.462+0000 7f0b56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:11.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.470+0000 7f0b56ffd640 10 client.?.objecter ms_handle_connect 0x7f0b6006c2d0 2024-03-25T14:25:11.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.470+0000 7f0b56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:11.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.477+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185217692 0 0) 0x7f0b60148000 con 0x7f0b6006c2d0 2024-03-25T14:25:11.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.477+0000 7f0b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:25:11.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.477+0000 7f0b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:11.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.477+0000 7f0b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:11.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.477+0000 7f0b56ffd640 10 monclient(hunting): my global_id is 4464 2024-03-25T14:25:11.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.477+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40004eb0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.487+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2337701119 0 0) 0x7f0b40004eb0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.487+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b60148000 con 0x7f0b6006c2d0 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.487+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0b500018c0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.487+0000 7f0b56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.487+0000 7f0b56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.487+0000 7f0b56ffd640 10 monclient(hunting): dump: 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:11.492 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:11.493 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:11.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.487+0000 7f0b56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1574593619 0 0) 0x7f0b60148000 con 0x7f0b6006c2d0 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 >> v1:172.21.15.63:6790/0 conn(0x7f0b6014aeb0 legacy=0x7f0b60148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 >> v1:172.21.15.153:6789/0 conn(0x7f0b6006c6b0 legacy=0x7f0b601486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 1 monclient: found mon.a 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b601db3c0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 10 monclient: _finish_auth 0 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:11.502 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b5f59e640 5 monclient: authenticate success, global_id 4464 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b5f59e640 10 monclient: _renew_subs 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.497+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b60149b90 con 0x7f0b6006c2d0 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 10 monclient: _renew_subs 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b601e2fe0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 1 librados: init done 2024-03-25T14:25:11.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:11.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:11.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:11.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.498+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b601431e0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b500035f0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0b500039c0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b56ffd640 10 monclient: dump: 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:11.791 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:11.792 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:11.792 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:11.792 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:11.792 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:11.792 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:11.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b5d59a640 4 set_mon_vals no callback set 2024-03-25T14:25:11.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.787+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f0b50003be0 con 0x7f0b6006c2d0 2024-03-25T14:25:11.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.788+0000 7f0b57fff640 10 monclient: get_auth_request con 0x7f0b4006dd70 auth_method 0 2024-03-25T14:25:11.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.788+0000 7f0b56ffd640 10 client.4464.objecter ms_handle_connect 0x7f0b4006dd70 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f0b50053570 con 0x7f0b6006c2d0 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 10 client.4464.objecter ms_dispatch 0x7f0b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b5004f080 con 0x7f0b6006c2d0 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:11.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:11.797+0000 7f0b56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:12.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.031+0000 7f0b5f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:12.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.031+0000 7f0b5f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:12.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.031+0000 7f0b5f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:12.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.031+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b600627b0 con 0x7f0b6006c2d0 2024-03-25T14:25:12.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.070+0000 7f0b56ffd640 1 -- 172.21.15.63:0/4092378049 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f0b50050fb0 con 0x7f0b6006c2d0 2024-03-25T14:25:12.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.070+0000 7f0b56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:12.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.070+0000 7f0b56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:12.075 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:12.075 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:12.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 10 librados: watch_flush enter 2024-03-25T14:25:12.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 10 librados: watch_flush exit 2024-03-25T14:25:12.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-25T14:25:12.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 10 client.4464.objecter successfully canceled tick 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 >> v1:172.21.15.153:6800/1688255552 conn(0x7f0b4006dd70 legacy=0x7f0b40070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 10 monclient: shutdown 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 >> v1:172.21.15.63:6789/0 conn(0x7f0b6006c2d0 legacy=0x7f0b601dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 shutdown_connections 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 >> 172.21.15.63:0/4092378049 conn(0x7f0b600b5b90 msgr2=0x7f0b600b5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 shutdown_connections 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 1 -- 172.21.15.63:0/4092378049 wait complete. 2024-03-25T14:25:12.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:12.073+0000 7f0b5f59e640 1 librados: shutdown 2024-03-25T14:25:12.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:18.896 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:19.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.009+0000 7f1c3fab3640 10 monclient: get_monmap_and_config 2024-03-25T14:25:19.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.009+0000 7f1c3fab3640 10 monclient: build_initial_monmap 2024-03-25T14:25:19.013 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.009+0000 7f1c3fab3640 10 monclient: monmap: 2024-03-25T14:25:19.013 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:19.013 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:19.013 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:19.010401+0000 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:19.010401+0000 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:19.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.009+0000 7f1c3fab3640 1 Processor -- start 2024-03-25T14:25:19.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 1 -- start start 2024-03-25T14:25:19.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 10 monclient: init 2024-03-25T14:25:19.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:19.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 10 monclient(hunting): picked mon.noname-c con 0x7f1c380a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3813f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 10 monclient(hunting): picked mon.noname-b con 0x7f1c381430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c38142520 con 0x7f1c3813f710 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c380b28a0 con 0x7f1c380a98f0 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.010+0000 7f1c3fab3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c380b2a30 con 0x7f1c381430c0 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.011+0000 7f1c3fab3640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:19.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.011+0000 7f1c3fab3640 10 monclient(hunting): authenticate will time out at 1682.155526s 2024-03-25T14:25:19.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.021+0000 7f1c3d84f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1c380a98f0 0x7f1c380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:50810/0 (socket says 172.21.15.63:50810) 2024-03-25T14:25:19.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.021+0000 7f1c3d84f640 1 -- 172.21.15.63:0/3545370230 learned_addr learned my addr 172.21.15.63:0/3545370230 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:19.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.031+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514723202 0 0) 0x7f1c380b28a0 con 0x7f1c380a98f0 2024-03-25T14:25:19.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.031+0000 7f1c3c84d640 10 monclient(hunting): _negotiate 2024-03-25T14:25:19.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.031+0000 7f1c3c84d640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:19.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.031+0000 7f1c3c84d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:19.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.031+0000 7f1c3c84d640 10 monclient(hunting): my global_id is 4468 2024-03-25T14:25:19.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.031+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c24003520 con 0x7f1c380a98f0 2024-03-25T14:25:19.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.047+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742304604 0 0) 0x7f1c380b2a30 con 0x7f1c381430c0 2024-03-25T14:25:19.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.047+0000 7f1c3c84d640 10 monclient(hunting): _negotiate 2024-03-25T14:25:19.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.047+0000 7f1c3c84d640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:19.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.047+0000 7f1c3c84d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:19.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.047+0000 7f1c3c84d640 10 monclient(hunting): my global_id is 4448 2024-03-25T14:25:19.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.047+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c380b28a0 con 0x7f1c381430c0 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1921274671 0 0) 0x7f1c24003520 con 0x7f1c380a98f0 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 >> v1:172.21.15.63:6790/0 conn(0x7f1c381430c0 legacy=0x7f1c3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 >> v1:172.21.15.63:6789/0 conn(0x7f1c3813f710 legacy=0x7f1c381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 10 monclient: _finish_hunting 0 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 1 monclient: found mon.noname-c 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c380b44c0 con 0x7f1c380a98f0 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 10 monclient: _finish_auth 0 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 10 monclient: _check_auth_tickets 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:19.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c34001690 con 0x7f1c380a98f0 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3c84d640 10 monclient: dump: 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:19.054 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:19.055 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:19.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3fab3640 5 monclient: authenticate success, global_id 4468 2024-03-25T14:25:19.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.048+0000 7f1c3fab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:19.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c340017f0 con 0x7f1c380a98f0 2024-03-25T14:25:19.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3c84d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:19.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3c84d640 1 -- 172.21.15.63:0/3545370230 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c34001c00 con 0x7f1c380a98f0 2024-03-25T14:25:19.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3c84d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:19.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3c84d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3c84d640 10 monclient: dump: 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3fab3640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:19.070 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.065+0000 7f1c3fab3640 4 set_mon_vals no callback set 2024-03-25T14:25:19.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.066+0000 7f1c3fab3640 10 monclient: shutdown 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.066+0000 7f1c3fab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.066+0000 7f1c3fab3640 1 -- 172.21.15.63:0/3545370230 >> v1:172.21.15.153:6789/0 conn(0x7f1c380a98f0 legacy=0x7f1c380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.066+0000 7f1c3fab3640 1 -- 172.21.15.63:0/3545370230 shutdown_connections 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.066+0000 7f1c3fab3640 1 -- 172.21.15.63:0/3545370230 >> 172.21.15.63:0/3545370230 conn(0x7f1c380b7ea0 msgr2=0x7f1c380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.066+0000 7f1c3fab3640 1 -- 172.21.15.63:0/3545370230 shutdown_connections 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.066+0000 7f1c3fab3640 1 -- 172.21.15.63:0/3545370230 wait complete. 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 10 monclient: build_initial_monmap 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 10 monclient: monmap: 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:19.073 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 1 librados: starting msgr at 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 1 librados: starting objecter 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 1 Processor -- start 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 1 -- start start 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 1 librados: setting wanted keys 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 1 librados: calling monclient init 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.067+0000 7f1c3fab3640 10 monclient: init 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:19.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 10 monclient(hunting): picked mon.noname-b con 0x7f1c380a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3813f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 10 monclient(hunting): picked mon.noname-c con 0x7f1c381430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c38136ca0 con 0x7f1c3813f710 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c38136e30 con 0x7f1c380a98f0 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c381377d0 con 0x7f1c381430c0 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:19.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.068+0000 7f1c3fab3640 10 monclient(hunting): authenticate will time out at 1682.212834s 2024-03-25T14:25:19.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.077+0000 7f1c3d04e640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f1c3813f710 0x7f1c381390a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:52064/0 (socket says 172.21.15.63:52064) 2024-03-25T14:25:19.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.077+0000 7f1c3d04e640 1 -- 172.21.15.63:0/272435337 learned_addr learned my addr 172.21.15.63:0/272435337 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.082+0000 7f1c2e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c3813f710 2024-03-25T14:25:19.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.082+0000 7f1c2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:19.090 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c381430c0 2024-03-25T14:25:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323569255 0 0) 0x7f1c38136ca0 con 0x7f1c3813f710 2024-03-25T14:25:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:25:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:19.091 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:19.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 10 monclient(hunting): my global_id is 4470 2024-03-25T14:25:19.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.087+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c1c004eb0 con 0x7f1c3813f710 2024-03-25T14:25:19.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.088+0000 7f1c2e7fc640 10 client.?.objecter ms_handle_connect 0x7f1c380a98f0 2024-03-25T14:25:19.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.088+0000 7f1c2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:19.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.093+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420930714 0 0) 0x7f1c381377d0 con 0x7f1c381430c0 2024-03-25T14:25:19.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.093+0000 7f1c2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:25:19.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.093+0000 7f1c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.093+0000 7f1c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.093+0000 7f1c2e7fc640 10 monclient(hunting): my global_id is 4451 2024-03-25T14:25:19.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.093+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c38136ca0 con 0x7f1c381430c0 2024-03-25T14:25:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.096+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2182852303 0 0) 0x7f1c1c004eb0 con 0x7f1c3813f710 2024-03-25T14:25:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.096+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c381377d0 con 0x7f1c3813f710 2024-03-25T14:25:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.096+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c280019d0 con 0x7f1c3813f710 2024-03-25T14:25:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.096+0000 7f1c2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.096+0000 7f1c2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:25:19.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.096+0000 7f1c2e7fc640 10 monclient(hunting): dump: 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:19.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.096+0000 7f1c2e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:19.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.101+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156729478 0 0) 0x7f1c38136e30 con 0x7f1c380a98f0 2024-03-25T14:25:19.104 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.101+0000 7f1c2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:25:19.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.101+0000 7f1c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:19.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.101+0000 7f1c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:19.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.101+0000 7f1c2e7fc640 10 monclient(hunting): my global_id is 4471 2024-03-25T14:25:19.105 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.101+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c1c004eb0 con 0x7f1c380a98f0 2024-03-25T14:25:19.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 67094527 0 0) 0x7f1c381377d0 con 0x7f1c3813f710 2024-03-25T14:25:19.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 >> v1:172.21.15.63:6790/0 conn(0x7f1c381430c0 legacy=0x7f1c38133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:19.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 >> v1:172.21.15.153:6789/0 conn(0x7f1c380a98f0 legacy=0x7f1c381370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:19.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:25:19.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 1 monclient: found mon.a 2024-03-25T14:25:19.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:19.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38137960 con 0x7f1c3813f710 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 10 monclient: _finish_auth 0 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c3fab3640 5 monclient: authenticate success, global_id 4470 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c3fab3640 10 monclient: _renew_subs 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c3fab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.107+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c381397b0 con 0x7f1c3813f710 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.108+0000 7f1c3fab3640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.108+0000 7f1c3fab3640 10 monclient: _renew_subs 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.108+0000 7f1c3fab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.108+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c381338f0 con 0x7f1c3813f710 2024-03-25T14:25:19.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.108+0000 7f1c3fab3640 1 librados: init done 2024-03-25T14:25:19.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.109+0000 7f1c3fab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:19.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.109+0000 7f1c3fab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:19.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.109+0000 7f1c3fab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:19.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.109+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c38136620 con 0x7f1c3813f710 2024-03-25T14:25:19.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.419+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c28003610 con 0x7f1c3813f710 2024-03-25T14:25:19.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.419+0000 7f1c2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:19.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.419+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c28003800 con 0x7f1c3813f710 2024-03-25T14:25:19.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.419+0000 7f1c2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:19.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.419+0000 7f1c2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:19.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.419+0000 7f1c2e7fc640 10 monclient: dump: 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:19.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.419+0000 7f1c2ffff640 4 set_mon_vals no callback set 2024-03-25T14:25:19.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.420+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f1c28003a20 con 0x7f1c3813f710 2024-03-25T14:25:19.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.421+0000 7f1c3d84f640 10 monclient: get_auth_request con 0x7f1c1c06de00 auth_method 0 2024-03-25T14:25:19.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.421+0000 7f1c2e7fc640 10 client.4470.objecter ms_handle_connect 0x7f1c1c06de00 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f1c28054680 con 0x7f1c3813f710 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 10 client.4470.objecter ms_dispatch 0x7f1c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c28050190 con 0x7f1c3813f710 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.427+0000 7f1c2e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:19.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.652+0000 7f1c3fab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:19.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.652+0000 7f1c3fab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:19.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.652+0000 7f1c3fab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:19.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.652+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c380627b0 con 0x7f1c3813f710 2024-03-25T14:25:19.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.683+0000 7f1c2e7fc640 1 -- 172.21.15.63:0/272435337 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f1c280520c0 con 0x7f1c3813f710 2024-03-25T14:25:19.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.683+0000 7f1c2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:19.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.683+0000 7f1c2e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:19.687 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:19.687 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:19.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 10 librados: watch_flush enter 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 10 librados: watch_flush exit 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 10 client.4470.objecter successfully canceled tick 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 >> v1:172.21.15.153:6800/1688255552 conn(0x7f1c1c06de00 legacy=0x7f1c1c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 10 monclient: shutdown 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 >> v1:172.21.15.63:6789/0 conn(0x7f1c3813f710 legacy=0x7f1c381390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 shutdown_connections 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 >> 172.21.15.63:0/272435337 conn(0x7f1c380b7ea0 msgr2=0x7f1c38074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 shutdown_connections 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 1 -- 172.21.15.63:0/272435337 wait complete. 2024-03-25T14:25:19.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:19.685+0000 7f1c3fab3640 1 librados: shutdown 2024-03-25T14:25:19.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:26.508 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.620+0000 7fa22a196640 10 monclient: get_monmap_and_config 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.620+0000 7fa22a196640 10 monclient: build_initial_monmap 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.620+0000 7fa22a196640 10 monclient: monmap: 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:26.621313+0000 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:26.621313+0000 2024-03-25T14:25:26.624 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:26.625 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:26.625 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:26.625 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:26.625 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:26.625 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 1 Processor -- start 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 1 -- start start 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient: init 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient(hunting): picked mon.noname-b con 0x7fa2240a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient(hunting): picked mon.noname-a con 0x7fa22413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient(hunting): picked mon.noname-c con 0x7fa2241430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:26.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa224142520 con 0x7fa22413f710 2024-03-25T14:25:26.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240b28a0 con 0x7fa2241430c0 2024-03-25T14:25:26.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240b2a30 con 0x7fa2240a98f0 2024-03-25T14:25:26.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:26.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.621+0000 7fa22a196640 10 monclient(hunting): authenticate will time out at 1689.766510s 2024-03-25T14:25:26.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.625+0000 7fa223fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa2241430c0 0x7fa22414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:35752/0 (socket says 172.21.15.63:35752) 2024-03-25T14:25:26.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.625+0000 7fa223fff640 1 -- 172.21.15.63:0/3725487946 learned_addr learned my addr 172.21.15.63:0/3725487946 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:26.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.630+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886068282 0 0) 0x7fa2240b28a0 con 0x7fa2241430c0 2024-03-25T14:25:26.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.630+0000 7fa2227fc640 10 monclient(hunting): _negotiate 2024-03-25T14:25:26.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.630+0000 7fa2227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:26.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.630+0000 7fa2227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:26.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.630+0000 7fa2227fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T14:25:26.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.630+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c003520 con 0x7fa2241430c0 2024-03-25T14:25:26.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2980381439 0 0) 0x7fa20c003520 con 0x7fa2241430c0 2024-03-25T14:25:26.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 >> v1:172.21.15.63:6790/0 conn(0x7fa2240a98f0 legacy=0x7fa2240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:25:26.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 >> v1:172.21.15.63:6789/0 conn(0x7fa22413f710 legacy=0x7fa2241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 10 monclient: _finish_hunting 0 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 1 monclient: found mon.noname-c 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240b44c0 con 0x7fa2241430c0 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 10 monclient: _finish_auth 0 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 10 monclient: _check_auth_tickets 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa21c001a10 con 0x7fa2241430c0 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa2227fc640 10 monclient: dump: 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:26.644 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa22a196640 5 monclient: authenticate success, global_id 4474 2024-03-25T14:25:26.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.639+0000 7fa22a196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:26.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa21c001be0 con 0x7fa2241430c0 2024-03-25T14:25:26.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa2227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa2227fc640 1 -- 172.21.15.63:0/3725487946 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa21c002030 con 0x7fa2241430c0 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa2227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa2227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa2227fc640 10 monclient: dump: 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:26.651 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:26.652 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:26.652 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:26.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa22a196640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:26.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.646+0000 7fa22a196640 4 set_mon_vals no callback set 2024-03-25T14:25:26.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.647+0000 7fa22a196640 10 monclient: shutdown 2024-03-25T14:25:26.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.647+0000 7fa22a196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:26.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.647+0000 7fa22a196640 1 -- 172.21.15.63:0/3725487946 >> v1:172.21.15.153:6789/0 conn(0x7fa2241430c0 legacy=0x7fa22414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:26.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.647+0000 7fa22a196640 1 -- 172.21.15.63:0/3725487946 shutdown_connections 2024-03-25T14:25:26.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.647+0000 7fa22a196640 1 -- 172.21.15.63:0/3725487946 >> 172.21.15.63:0/3725487946 conn(0x7fa2240b7ea0 msgr2=0x7fa2240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:26.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.648+0000 7fa22a196640 1 -- 172.21.15.63:0/3725487946 shutdown_connections 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.648+0000 7fa22a196640 1 -- 172.21.15.63:0/3725487946 wait complete. 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.648+0000 7fa22a196640 10 monclient: build_initial_monmap 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.648+0000 7fa22a196640 10 monclient: monmap: 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:26.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.649+0000 7fa22a196640 1 librados: starting msgr at 2024-03-25T14:25:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.649+0000 7fa22a196640 1 librados: starting objecter 2024-03-25T14:25:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.649+0000 7fa22a196640 1 Processor -- start 2024-03-25T14:25:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.649+0000 7fa22a196640 1 -- start start 2024-03-25T14:25:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.649+0000 7fa22a196640 1 librados: setting wanted keys 2024-03-25T14:25:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.649+0000 7fa22a196640 1 librados: calling monclient init 2024-03-25T14:25:26.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.649+0000 7fa22a196640 10 monclient: init 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 10 monclient(hunting): picked mon.noname-b con 0x7fa2240a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 10 monclient(hunting): picked mon.noname-a con 0x7fa22413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 10 monclient(hunting): picked mon.noname-c con 0x7fa2241430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa224137fc0 con 0x7fa22413f710 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2241d82b0 con 0x7fa2240a98f0 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa224139950 con 0x7fa2241430c0 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:26.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.650+0000 7fa22a196640 10 monclient(hunting): authenticate will time out at 1689.795196s 2024-03-25T14:25:26.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.664+0000 7fa223fff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fa2241430c0 0x7fa2241e1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:35612/0 (socket says 172.21.15.63:35612) 2024-03-25T14:25:26.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.664+0000 7fa223fff640 1 -- 172.21.15.63:0/986519651 learned_addr learned my addr 172.21.15.63:0/986519651 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:26.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.665+0000 7fa20bfff640 10 client.?.objecter ms_handle_connect 0x7fa2241430c0 2024-03-25T14:25:26.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.666+0000 7fa20bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:26.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.669+0000 7fa20bfff640 10 client.?.objecter ms_handle_connect 0x7fa22413f710 2024-03-25T14:25:26.673 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.669+0000 7fa20bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:26.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.681+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199290790 0 0) 0x7fa224137fc0 con 0x7fa22413f710 2024-03-25T14:25:26.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:25:26.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:26.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:26.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): my global_id is 4476 2024-03-25T14:25:26.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa210004eb0 con 0x7fa22413f710 2024-03-25T14:25:26.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493483958 0 0) 0x7fa224139950 con 0x7fa2241430c0 2024-03-25T14:25:26.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): _negotiate 2024-03-25T14:25:26.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:26.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:26.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 10 monclient(hunting): my global_id is 4454 2024-03-25T14:25:26.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.682+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa224137fc0 con 0x7fa2241430c0 2024-03-25T14:25:26.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.688+0000 7fa20bfff640 10 client.?.objecter ms_handle_connect 0x7fa2240a98f0 2024-03-25T14:25:26.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.688+0000 7fa20bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:26.697 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.693+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3749863544 0 0) 0x7fa224137fc0 con 0x7fa2241430c0 2024-03-25T14:25:26.697 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.693+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa224139950 con 0x7fa2241430c0 2024-03-25T14:25:26.697 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.693+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa21c001ef0 con 0x7fa2241430c0 2024-03-25T14:25:26.697 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.693+0000 7fa20bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:26.697 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.693+0000 7fa20bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:25:26.697 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.693+0000 7fa20bfff640 10 monclient(hunting): dump: 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:26.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.693+0000 7fa20bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:26.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.696+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2372021685 0 0) 0x7fa210004eb0 con 0x7fa22413f710 2024-03-25T14:25:26.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.696+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa224137fc0 con 0x7fa22413f710 2024-03-25T14:25:26.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.696+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa218001930 con 0x7fa22413f710 2024-03-25T14:25:26.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.696+0000 7fa20bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:26.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.696+0000 7fa20bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.696+0000 7fa20bfff640 10 monclient(hunting): dump: 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:26.701 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.705+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4238661637 0 0) 0x7fa224139950 con 0x7fa2241430c0 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 >> v1:172.21.15.153:6789/0 conn(0x7fa2240a98f0 legacy=0x7fa2241385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 >> v1:172.21.15.63:6789/0 conn(0x7fa22413f710 legacy=0x7fa2241d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 10 monclient: _finish_hunting 0 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 1 monclient: found mon.c 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:26.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2241350d0 con 0x7fa2241430c0 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 10 monclient: _finish_auth 0 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 10 monclient: _check_auth_tickets 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa20bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 5 monclient: authenticate success, global_id 4454 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 10 monclient: _renew_subs 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa224135380 con 0x7fa2241430c0 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 10 monclient: _renew_subs 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa224137a90 con 0x7fa2241430c0 2024-03-25T14:25:26.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.706+0000 7fa22a196640 1 librados: init done 2024-03-25T14:25:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.707+0000 7fa22a196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.707+0000 7fa22a196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.707+0000 7fa22a196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:26.712 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:26.707+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa224137d90 con 0x7fa2241430c0 2024-03-25T14:25:27.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.021+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa21c0020f0 con 0x7fa2241430c0 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.021+0000 7fa20bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.021+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa21c004e10 con 0x7fa2241430c0 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.021+0000 7fa20bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.021+0000 7fa20bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.021+0000 7fa20bfff640 10 monclient: dump: 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:27.026 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:27.027 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:27.027 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:27.027 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:27.027 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:27.027 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:27.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.021+0000 7fa221ffb640 4 set_mon_vals no callback set 2024-03-25T14:25:27.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.022+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fa21c01ade0 con 0x7fa2241430c0 2024-03-25T14:25:27.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.023+0000 7fa2237fe640 10 monclient: get_auth_request con 0x7fa21006ded0 auth_method 0 2024-03-25T14:25:27.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.023+0000 7fa20bfff640 10 client.4454.objecter ms_handle_connect 0x7fa21006ded0 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.030+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fa21c0549b0 con 0x7fa2241430c0 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.030+0000 7fa20bfff640 10 client.4454.objecter ms_dispatch 0x7fa224000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.030+0000 7fa20bfff640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.030+0000 7fa20bfff640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.030+0000 7fa20bfff640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.031+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa21c0504c0 con 0x7fa2241430c0 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.031+0000 7fa20bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:27.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.031+0000 7fa20bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:27.114 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:05.627 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.128 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:05.640 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.161 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:05.673 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.194 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:05.654 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.198 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:05.658 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.200 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:05.713 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.209 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:05.722 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.228 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:05.688 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.256 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:05.768 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:27.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.254+0000 7fa22a196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:27.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.254+0000 7fa22a196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:27.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.254+0000 7fa22a196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:27.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.254+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2240627b0 con 0x7fa2241430c0 2024-03-25T14:25:27.299 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.295+0000 7fa20bfff640 1 -- 172.21.15.63:0/986519651 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fa21c0523f0 con 0x7fa2241430c0 2024-03-25T14:25:27.299 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.295+0000 7fa20bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:27.299 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.295+0000 7fa20bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:27.299 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:27.299 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:27.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 10 librados: watch_flush enter 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 10 librados: watch_flush exit 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 10 client.4454.objecter successfully canceled tick 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 >> v1:172.21.15.153:6800/1688255552 conn(0x7fa21006ded0 legacy=0x7fa210070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 10 monclient: shutdown 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 >> v1:172.21.15.63:6790/0 conn(0x7fa2241430c0 legacy=0x7fa2241e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 shutdown_connections 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 >> 172.21.15.63:0/986519651 conn(0x7fa2240b7ea0 msgr2=0x7fa224074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 shutdown_connections 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 1 -- 172.21.15.63:0/986519651 wait complete. 2024-03-25T14:25:27.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:27.297+0000 7fa22a196640 1 librados: shutdown 2024-03-25T14:25:27.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:34.119 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:34.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.227+0000 7f625e0c8640 10 monclient: get_monmap_and_config 2024-03-25T14:25:34.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.227+0000 7f625e0c8640 10 monclient: build_initial_monmap 2024-03-25T14:25:34.231 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.227+0000 7f625e0c8640 10 monclient: monmap: 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:34.228631+0000 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:34.228631+0000 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:34.232 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.228+0000 7f625e0c8640 1 Processor -- start 2024-03-25T14:25:34.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.228+0000 7f625e0c8640 1 -- start start 2024-03-25T14:25:34.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.228+0000 7f625e0c8640 10 monclient: init 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f62580b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f62580b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f625814be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580af940 con 0x7f625814be90 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580b0a70 con 0x7f62580b4d60 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580b0d50 con 0x7f62580b1730 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:34.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.229+0000 7f625e0c8640 10 monclient(hunting): authenticate will time out at 1697.373846s 2024-03-25T14:25:34.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.243+0000 7f6257fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f625814be90 0x7f625814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:41668/0 (socket says 172.21.15.63:41668) 2024-03-25T14:25:34.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.243+0000 7f6257fff640 1 -- 172.21.15.63:0/3935225002 learned_addr learned my addr 172.21.15.63:0/3935225002 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:34.265 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.261+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937395266 0 0) 0x7f62580af940 con 0x7f625814be90 2024-03-25T14:25:34.265 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.261+0000 7f62567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:25:34.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.261+0000 7f62567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:34.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.261+0000 7f62567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:34.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.262+0000 7f62567fc640 10 monclient(hunting): my global_id is 4479 2024-03-25T14:25:34.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.262+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6240003520 con 0x7f625814be90 2024-03-25T14:25:34.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.272+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690065179 0 0) 0x7f62580b0a70 con 0x7f62580b4d60 2024-03-25T14:25:34.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.272+0000 7f62567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:25:34.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.272+0000 7f62567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:34.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.272+0000 7f62567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:34.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.272+0000 7f62567fc640 10 monclient(hunting): my global_id is 4480 2024-03-25T14:25:34.276 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.272+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62580af940 con 0x7f62580b4d60 2024-03-25T14:25:34.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1506059880 0 0) 0x7f6240003520 con 0x7f625814be90 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 >> v1:172.21.15.63:6790/0 conn(0x7f62580b1730 legacy=0x7f62580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 >> v1:172.21.15.153:6789/0 conn(0x7f62580b4d60 legacy=0x7f625813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 10 monclient: _finish_hunting 0 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 1 monclient: found mon.noname-a 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580b1030 con 0x7f625814be90 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 10 monclient: _finish_auth 0 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 10 monclient: _check_auth_tickets 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:34.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6250001a10 con 0x7f625814be90 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f62567fc640 10 monclient: dump: 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f625e0c8640 5 monclient: authenticate success, global_id 4479 2024-03-25T14:25:34.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.277+0000 7f625e0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.299+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6250001b70 con 0x7f625814be90 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.299+0000 7f62567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.299+0000 7f62567fc640 1 -- 172.21.15.63:0/3935225002 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6250001f80 con 0x7f625814be90 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.299+0000 7f62567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.299+0000 7f62567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.299+0000 7f62567fc640 10 monclient: dump: 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.304 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:34.305 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:34.305 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:34.305 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:34.305 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:34.305 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.300+0000 7f625e0c8640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:34.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.300+0000 7f625e0c8640 4 set_mon_vals no callback set 2024-03-25T14:25:34.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.300+0000 7f625e0c8640 10 monclient: shutdown 2024-03-25T14:25:34.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.300+0000 7f625e0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.300+0000 7f625e0c8640 1 -- 172.21.15.63:0/3935225002 >> v1:172.21.15.63:6789/0 conn(0x7f625814be90 legacy=0x7f625814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.301+0000 7f625e0c8640 1 -- 172.21.15.63:0/3935225002 shutdown_connections 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.301+0000 7f625e0c8640 1 -- 172.21.15.63:0/3935225002 >> 172.21.15.63:0/3935225002 conn(0x7f62580617a0 msgr2=0x7f6258061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.301+0000 7f625e0c8640 1 -- 172.21.15.63:0/3935225002 shutdown_connections 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.301+0000 7f625e0c8640 1 -- 172.21.15.63:0/3935225002 wait complete. 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.301+0000 7f625e0c8640 10 monclient: build_initial_monmap 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.301+0000 7f625e0c8640 10 monclient: monmap: 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:34.307 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.302+0000 7f625e0c8640 1 librados: starting msgr at 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.302+0000 7f625e0c8640 1 librados: starting objecter 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.302+0000 7f625e0c8640 1 Processor -- start 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.302+0000 7f625e0c8640 1 -- start start 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.302+0000 7f625e0c8640 1 librados: setting wanted keys 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.302+0000 7f625e0c8640 1 librados: calling monclient init 2024-03-25T14:25:34.308 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.302+0000 7f625e0c8640 10 monclient: init 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f62580b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f62580b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f625814be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6258134a00 con 0x7f625814be90 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6258139050 con 0x7f62580b1730 2024-03-25T14:25:34.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625813a330 con 0x7f62580b4d60 2024-03-25T14:25:34.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:34.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.303+0000 7f625e0c8640 10 monclient(hunting): authenticate will time out at 1697.447820s 2024-03-25T14:25:34.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.314+0000 7f62577fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f62580b1730 0x7f625813ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:47586/0 (socket says 172.21.15.63:47586) 2024-03-25T14:25:34.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.314+0000 7f6256ffd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f62580b4d60 0x7f6258133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:35400/0 (socket says 172.21.15.63:35400) 2024-03-25T14:25:34.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.314+0000 7f62577fe640 1 -- 172.21.15.63:0/1028422222 learned_addr learned my addr 172.21.15.63:0/1028422222 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:34.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.315+0000 7f623ffff640 10 client.?.objecter ms_handle_connect 0x7f625814be90 2024-03-25T14:25:34.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.315+0000 7f623ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:34.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.316+0000 7f623ffff640 10 client.?.objecter ms_handle_connect 0x7f62580b4d60 2024-03-25T14:25:34.320 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.316+0000 7f623ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:34.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.318+0000 7f623ffff640 10 client.?.objecter ms_handle_connect 0x7f62580b1730 2024-03-25T14:25:34.321 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.318+0000 7f623ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:34.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.326+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144938459 0 0) 0x7f625813a330 con 0x7f62580b4d60 2024-03-25T14:25:34.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.326+0000 7f623ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:25:34.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.326+0000 7f623ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:34.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.326+0000 7f623ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:34.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.326+0000 7f623ffff640 10 monclient(hunting): my global_id is 4460 2024-03-25T14:25:34.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.326+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6244004eb0 con 0x7f62580b4d60 2024-03-25T14:25:34.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.327+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339236464 0 0) 0x7f6258139050 con 0x7f62580b1730 2024-03-25T14:25:34.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.327+0000 7f623ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:25:34.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.327+0000 7f623ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:34.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.327+0000 7f623ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:34.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.327+0000 7f623ffff640 10 monclient(hunting): my global_id is 4483 2024-03-25T14:25:34.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.327+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625813a330 con 0x7f62580b1730 2024-03-25T14:25:34.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.334+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451047284 0 0) 0x7f6258134a00 con 0x7f625814be90 2024-03-25T14:25:34.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.334+0000 7f623ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:25:34.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.334+0000 7f623ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:34.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.334+0000 7f623ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:34.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.334+0000 7f623ffff640 10 monclient(hunting): my global_id is 4482 2024-03-25T14:25:34.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.334+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6258139050 con 0x7f625814be90 2024-03-25T14:25:34.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.352+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 712661137 0 0) 0x7f6244004eb0 con 0x7f62580b4d60 2024-03-25T14:25:34.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.352+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6258134a00 con 0x7f62580b4d60 2024-03-25T14:25:34.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.352+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f624c001910 con 0x7f62580b4d60 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.352+0000 7f623ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.352+0000 7f623ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.352+0000 7f623ffff640 10 monclient(hunting): dump: 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:34.357 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.358 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.352+0000 7f623ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.360+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41494713 0 0) 0x7f6258139050 con 0x7f625814be90 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.360+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6244004eb0 con 0x7f625814be90 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.360+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6250001e80 con 0x7f625814be90 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.360+0000 7f623ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.360+0000 7f623ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.360+0000 7f623ffff640 10 monclient(hunting): dump: 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.364 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:34.365 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:34.365 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:34.365 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:34.365 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:34.365 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.362+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2805280925 0 0) 0x7f625813a330 con 0x7f62580b1730 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.362+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6258139050 con 0x7f62580b1730 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.362+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f62480019f0 con 0x7f62580b1730 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.362+0000 7f623ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.362+0000 7f623ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.362+0000 7f623ffff640 10 monclient(hunting): dump: 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:34.366 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:34.367 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:34.367 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:34.367 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:34.367 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2919573471 0 0) 0x7f6258134a00 con 0x7f62580b4d60 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 >> v1:172.21.15.153:6789/0 conn(0x7f62580b1730 legacy=0x7f625813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 >> v1:172.21.15.63:6789/0 conn(0x7f625814be90 legacy=0x7f62581342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 1 monclient: found mon.c 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625813a610 con 0x7f62580b4d60 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 10 monclient: _finish_auth 0 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:25:34.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f623ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 5 monclient: authenticate success, global_id 4460 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 10 monclient: _renew_subs 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625813a8f0 con 0x7f62580b4d60 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 10 monclient: _renew_subs 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6258139730 con 0x7f62580b4d60 2024-03-25T14:25:34.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.370+0000 7f625e0c8640 1 librados: init done 2024-03-25T14:25:34.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.371+0000 7f625e0c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:34.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.371+0000 7f625e0c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:34.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.371+0000 7f625e0c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:34.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.371+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62581391e0 con 0x7f62580b4d60 2024-03-25T14:25:34.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.689+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f624c003610 con 0x7f62580b4d60 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.689+0000 7f623ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.689+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f624c003800 con 0x7f62580b4d60 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.689+0000 7f623ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.689+0000 7f623ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.689+0000 7f623ffff640 10 monclient: dump: 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:34.694 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:34.695 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:34.695 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:34.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.689+0000 7f6255ffb640 4 set_mon_vals no callback set 2024-03-25T14:25:34.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.690+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f624c003a20 con 0x7f62580b4d60 2024-03-25T14:25:34.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.691+0000 7f62577fe640 10 monclient: get_auth_request con 0x7f624406e0f0 auth_method 0 2024-03-25T14:25:34.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.691+0000 7f623ffff640 10 client.4460.objecter ms_handle_connect 0x7f624406e0f0 2024-03-25T14:25:34.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f624c054580 con 0x7f62580b4d60 2024-03-25T14:25:34.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 10 client.4460.objecter ms_dispatch 0x7f6258000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:34.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:34.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:34.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-25T14:25:34.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f624c050090 con 0x7f62580b4d60 2024-03-25T14:25:34.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:34.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.697+0000 7f623ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:34.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.944+0000 7f625e0c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:34.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.944+0000 7f625e0c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:34.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.944+0000 7f625e0c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:25:34.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.944+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62580627b0 con 0x7f62580b4d60 2024-03-25T14:25:34.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.973+0000 7f623ffff640 1 -- 172.21.15.63:0/1028422222 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f624c051fc0 con 0x7f62580b4d60 2024-03-25T14:25:34.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.973+0000 7f623ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:34.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.973+0000 7f623ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:34.978 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:34.978 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:34.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 10 librados: watch_flush enter 2024-03-25T14:25:34.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 10 librados: watch_flush exit 2024-03-25T14:25:34.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-25T14:25:34.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 10 client.4460.objecter successfully canceled tick 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 >> v1:172.21.15.153:6800/1688255552 conn(0x7f624406e0f0 legacy=0x7f62440705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 10 monclient: shutdown 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 >> v1:172.21.15.63:6790/0 conn(0x7f62580b4d60 legacy=0x7f6258133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 shutdown_connections 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 >> 172.21.15.63:0/1028422222 conn(0x7f62580617a0 msgr2=0x7f62580b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 shutdown_connections 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 1 -- 172.21.15.63:0/1028422222 wait complete. 2024-03-25T14:25:34.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:34.976+0000 7f625e0c8640 1 librados: shutdown 2024-03-25T14:25:34.992 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:41.800 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:41.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.910+0000 7f79e2e0e640 10 monclient: get_monmap_and_config 2024-03-25T14:25:41.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.910+0000 7f79e2e0e640 10 monclient: build_initial_monmap 2024-03-25T14:25:41.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.910+0000 7f79e2e0e640 10 monclient: monmap: 2024-03-25T14:25:41.915 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:41.915 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:41.915 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:41.912038+0000 2024-03-25T14:25:41.915 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:41.912038+0000 2024-03-25T14:25:41.916 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:41.916 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:41.916 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:41.916 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:41.916 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:41.916 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.911+0000 7f79e2e0e640 1 Processor -- start 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.911+0000 7f79e2e0e640 1 -- start start 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.911+0000 7f79e2e0e640 10 monclient: init 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 10 monclient(hunting): picked mon.noname-b con 0x7f79dc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 10 monclient(hunting): picked mon.noname-c con 0x7f79dc13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 10 monclient(hunting): picked mon.noname-a con 0x7f79dc1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc142520 con 0x7f79dc1430c0 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc0b28a0 con 0x7f79dc13f710 2024-03-25T14:25:41.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc0b2a30 con 0x7f79dc0a98f0 2024-03-25T14:25:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:41.918 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.912+0000 7f79e2e0e640 10 monclient(hunting): authenticate will time out at 1705.057214s 2024-03-25T14:25:41.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.920+0000 7f79e0baa640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f79dc0a98f0 0x7f79dc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:41090/0 (socket says 172.21.15.63:41090) 2024-03-25T14:25:41.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.920+0000 7f79e0baa640 1 -- 172.21.15.63:0/1199779086 learned_addr learned my addr 172.21.15.63:0/1199779086 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:41.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.930+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855276092 0 0) 0x7f79dc0b2a30 con 0x7f79dc0a98f0 2024-03-25T14:25:41.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.930+0000 7f79db7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:25:41.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.930+0000 7f79db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:41.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.930+0000 7f79db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:41.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.930+0000 7f79db7fe640 10 monclient(hunting): my global_id is 4463 2024-03-25T14:25:41.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.930+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79cc003520 con 0x7f79dc0a98f0 2024-03-25T14:25:41.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.938+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042304635 0 0) 0x7f79dc142520 con 0x7f79dc1430c0 2024-03-25T14:25:41.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.938+0000 7f79db7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:25:41.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.938+0000 7f79db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:41.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.938+0000 7f79db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:41.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.938+0000 7f79db7fe640 10 monclient(hunting): my global_id is 4485 2024-03-25T14:25:41.942 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.938+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79dc0b2a30 con 0x7f79dc1430c0 2024-03-25T14:25:41.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.948+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995544595 0 0) 0x7f79cc003520 con 0x7f79dc0a98f0 2024-03-25T14:25:41.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.948+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 >> v1:172.21.15.153:6789/0 conn(0x7f79dc13f710 legacy=0x7f79dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:41.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.948+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 >> v1:172.21.15.63:6789/0 conn(0x7f79dc1430c0 legacy=0x7f79dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:41.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:25:41.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 1 monclient: found mon.noname-b 2024-03-25T14:25:41.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:41.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0b44c0 con 0x7f79dc0a98f0 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 10 monclient: _finish_auth 0 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f79d0001690 con 0x7f79dc0a98f0 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79e2e0e640 5 monclient: authenticate success, global_id 4463 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79e2e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79db7fe640 10 monclient: dump: 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:41.954 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:41.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.949+0000 7f79e2e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79d00017f0 con 0x7f79dc0a98f0 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79db7fe640 1 -- 172.21.15.63:0/1199779086 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f79d0001c00 con 0x7f79dc0a98f0 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79e2e0e640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79e2e0e640 4 set_mon_vals no callback set 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:25:41.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.953+0000 7f79db7fe640 10 monclient: dump: 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:41.958 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:41.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 10 monclient: shutdown 2024-03-25T14:25:41.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:41.960 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 1 -- 172.21.15.63:0/1199779086 >> v1:172.21.15.63:6790/0 conn(0x7f79dc0a98f0 legacy=0x7f79dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 1 -- 172.21.15.63:0/1199779086 shutdown_connections 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 1 -- 172.21.15.63:0/1199779086 >> 172.21.15.63:0/1199779086 conn(0x7f79dc0b7ea0 msgr2=0x7f79dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 1 -- 172.21.15.63:0/1199779086 shutdown_connections 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 1 -- 172.21.15.63:0/1199779086 wait complete. 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 10 monclient: build_initial_monmap 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.954+0000 7f79e2e0e640 10 monclient: monmap: 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:41.961 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.955+0000 7f79e2e0e640 1 librados: starting msgr at 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.955+0000 7f79e2e0e640 1 librados: starting objecter 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.955+0000 7f79e2e0e640 1 Processor -- start 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.955+0000 7f79e2e0e640 1 -- start start 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.955+0000 7f79e2e0e640 1 librados: setting wanted keys 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.955+0000 7f79e2e0e640 1 librados: calling monclient init 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.955+0000 7f79e2e0e640 10 monclient: init 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 10 monclient(hunting): picked mon.noname-c con 0x7f79dc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:41.962 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 10 monclient(hunting): picked mon.noname-a con 0x7f79dc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:41.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 10 monclient(hunting): picked mon.noname-b con 0x7f79dc1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:41.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc1370a0 con 0x7f79dc13f710 2024-03-25T14:25:41.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc135090 con 0x7f79dc1430c0 2024-03-25T14:25:41.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79dc135220 con 0x7f79dc0a98f0 2024-03-25T14:25:41.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:41.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.956+0000 7f79e2e0e640 10 monclient(hunting): authenticate will time out at 1705.101099s 2024-03-25T14:25:41.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.963+0000 7f79e13ab640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f79dc1430c0 0x7f79dc1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:58500/0 (socket says 172.21.15.63:58500) 2024-03-25T14:25:41.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.963+0000 7f79e13ab640 1 -- 172.21.15.63:0/2805173176 learned_addr learned my addr 172.21.15.63:0/2805173176 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:41.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.964+0000 7f79d97fa640 10 client.?.objecter ms_handle_connect 0x7f79dc1430c0 2024-03-25T14:25:41.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.964+0000 7f79d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:41.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.970+0000 7f79d97fa640 10 client.?.objecter ms_handle_connect 0x7f79dc13f710 2024-03-25T14:25:41.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.970+0000 7f79d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:41.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.972+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481514207 0 0) 0x7f79dc135090 con 0x7f79dc1430c0 2024-03-25T14:25:41.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.972+0000 7f79d97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:25:41.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.972+0000 7f79d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:41.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.972+0000 7f79d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:41.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.972+0000 7f79d97fa640 10 monclient(hunting): my global_id is 4489 2024-03-25T14:25:41.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.973+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c0004eb0 con 0x7f79dc1430c0 2024-03-25T14:25:41.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.985+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240490324 0 0) 0x7f79dc1370a0 con 0x7f79dc13f710 2024-03-25T14:25:41.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.985+0000 7f79d97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:25:41.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.985+0000 7f79d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:41.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.985+0000 7f79d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:41.989 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.985+0000 7f79d97fa640 10 monclient(hunting): my global_id is 4488 2024-03-25T14:25:41.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.985+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79dc135090 con 0x7f79dc13f710 2024-03-25T14:25:41.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.989+0000 7f79d97fa640 10 client.?.objecter ms_handle_connect 0x7f79dc0a98f0 2024-03-25T14:25:41.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:41.989+0000 7f79d97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.001+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2761236942 0 0) 0x7f79c0004eb0 con 0x7f79dc1430c0 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.002+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79dc1370a0 con 0x7f79dc1430c0 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.002+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f79d4001c70 con 0x7f79dc1430c0 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.002+0000 7f79d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.002+0000 7f79d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.002+0000 7f79d97fa640 10 monclient(hunting): dump: 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:42.006 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:42.007 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:42.007 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:42.007 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:42.007 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:42.007 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:42.007 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:42.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.002+0000 7f79d97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:42.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.007+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2789367635 0 0) 0x7f79dc135090 con 0x7f79dc13f710 2024-03-25T14:25:42.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.007+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79c0004eb0 con 0x7f79dc13f710 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.007+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f79c80018f0 con 0x7f79dc13f710 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.007+0000 7f79d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.007+0000 7f79d97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.007+0000 7f79d97fa640 10 monclient(hunting): dump: 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:42.012 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:42.013 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:42.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.010+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994465623 0 0) 0x7f79dc135220 con 0x7f79dc0a98f0 2024-03-25T14:25:42.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.010+0000 7f79d97fa640 10 monclient(hunting): _negotiate 2024-03-25T14:25:42.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.010+0000 7f79d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:42.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.010+0000 7f79d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:42.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.010+0000 7f79d97fa640 10 monclient(hunting): my global_id is 4466 2024-03-25T14:25:42.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.010+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79dc135090 con 0x7f79dc0a98f0 2024-03-25T14:25:42.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2381231549 0 0) 0x7f79dc1370a0 con 0x7f79dc1430c0 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 >> v1:172.21.15.63:6790/0 conn(0x7f79dc0a98f0 legacy=0x7f79dc1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 >> v1:172.21.15.63:6789/0 conn(0x7f79dc13f710 legacy=0x7f79dc1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 10 monclient: _finish_hunting 0 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 1 monclient: found mon.b 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc1d5000 con 0x7f79dc1430c0 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 10 monclient: _finish_auth 0 2024-03-25T14:25:42.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 10 monclient: _check_auth_tickets 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 5 monclient: authenticate success, global_id 4489 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 10 monclient: _renew_subs 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc1d52b0 con 0x7f79dc1430c0 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 10 monclient: _renew_subs 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc135910 con 0x7f79dc1430c0 2024-03-25T14:25:42.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.025+0000 7f79e2e0e640 1 librados: init done 2024-03-25T14:25:42.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.026+0000 7f79e2e0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:42.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.026+0000 7f79e2e0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:42.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.026+0000 7f79e2e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:25:42.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.026+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc0627b0 con 0x7f79dc1430c0 2024-03-25T14:25:42.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.292+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79d4003990 con 0x7f79dc1430c0 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.292+0000 7f79d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.292+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f79d4003d60 con 0x7f79dc1430c0 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.292+0000 7f79d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.292+0000 7f79d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.292+0000 7f79d97fa640 10 monclient: dump: 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:42.297 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:42.298 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:42.298 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:42.298 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:42.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.292+0000 7f79daffd640 4 set_mon_vals no callback set 2024-03-25T14:25:42.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.293+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f79d4003f40 con 0x7f79dc1430c0 2024-03-25T14:25:42.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.294+0000 7f79e0baa640 10 monclient: get_auth_request con 0x7f79c006df50 auth_method 0 2024-03-25T14:25:42.299 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.294+0000 7f79d97fa640 10 client.4489.objecter ms_handle_connect 0x7f79c006df50 2024-03-25T14:25:42.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.298+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f79d4054b60 con 0x7f79dc1430c0 2024-03-25T14:25:42.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.298+0000 7f79d97fa640 10 client.4489.objecter ms_dispatch 0x7f79dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:42.302 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.298+0000 7f79d97fa640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:42.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.298+0000 7f79d97fa640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:42.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.298+0000 7f79d97fa640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T14:25:42.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.299+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79d4050670 con 0x7f79dc1430c0 2024-03-25T14:25:42.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.299+0000 7f79d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:42.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.299+0000 7f79d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:42.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.550+0000 7f79e2e0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:42.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.550+0000 7f79e2e0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:42.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.550+0000 7f79e2e0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:25:42.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.550+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79dc0008d0 con 0x7f79dc1430c0 2024-03-25T14:25:42.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.586+0000 7f79d97fa640 1 -- 172.21.15.63:0/2805173176 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f79d40525a0 con 0x7f79dc1430c0 2024-03-25T14:25:42.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.586+0000 7f79d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:42.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.586+0000 7f79d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:42.591 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:42.591 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 10 librados: watch_flush enter 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 10 librados: watch_flush exit 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 10 client.4489.objecter successfully canceled tick 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 >> v1:172.21.15.153:6800/1688255552 conn(0x7f79c006df50 legacy=0x7f79c0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 10 monclient: shutdown 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.589+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 >> v1:172.21.15.153:6789/0 conn(0x7f79dc1430c0 legacy=0x7f79dc1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.590+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 shutdown_connections 2024-03-25T14:25:42.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.590+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 >> 172.21.15.63:0/2805173176 conn(0x7f79dc0b7ea0 msgr2=0x7f79dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:42.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.590+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 shutdown_connections 2024-03-25T14:25:42.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.590+0000 7f79e2e0e640 1 -- 172.21.15.63:0/2805173176 wait complete. 2024-03-25T14:25:42.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:42.590+0000 7f79e2e0e640 1 librados: shutdown 2024-03-25T14:25:42.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:46.920 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:25.432 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:46.972 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:25.484 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:47.015 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:25.528 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:47.017 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:25.477 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:47.034 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:25.494 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:47.053 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:25.566 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:47.059 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:25.572 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:47.074 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:25.531 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:47.086 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:25.599 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:25:49.413 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:49.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.527+0000 7f05b035d640 10 monclient: get_monmap_and_config 2024-03-25T14:25:49.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.527+0000 7f05b035d640 10 monclient: build_initial_monmap 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.527+0000 7f05b035d640 10 monclient: monmap: 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:49.528555+0000 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:49.528555+0000 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:49.532 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 1 Processor -- start 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 1 -- start start 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient: init 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient(hunting): picked mon.noname-b con 0x7f05a80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient(hunting): picked mon.noname-a con 0x7f05a813f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:49.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient(hunting): picked mon.noname-c con 0x7f05a81430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:49.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a8142520 con 0x7f05a813f710 2024-03-25T14:25:49.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a80b28a0 con 0x7f05a81430c0 2024-03-25T14:25:49.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a80b2a30 con 0x7f05a80a98f0 2024-03-25T14:25:49.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:49.535 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.528+0000 7f05b035d640 10 monclient(hunting): authenticate will time out at 1712.673498s 2024-03-25T14:25:49.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.543+0000 7f05ad8f8640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f05a813f710 0x7f05a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:57680/0 (socket says 172.21.15.63:57680) 2024-03-25T14:25:49.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.543+0000 7f05ad8f8640 1 -- 172.21.15.63:0/1847710984 learned_addr learned my addr 172.21.15.63:0/1847710984 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:49.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.554+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361598134 0 0) 0x7f05a8142520 con 0x7f05a813f710 2024-03-25T14:25:49.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.555+0000 7f05ad0f7640 10 monclient(hunting): _negotiate 2024-03-25T14:25:49.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.555+0000 7f05ad0f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:49.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.555+0000 7f05ad0f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:49.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.555+0000 7f05ad0f7640 10 monclient(hunting): my global_id is 4491 2024-03-25T14:25:49.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.555+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0594003520 con 0x7f05a813f710 2024-03-25T14:25:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.556+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832696962 0 0) 0x7f05a80b2a30 con 0x7f05a80a98f0 2024-03-25T14:25:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.556+0000 7f05ad0f7640 10 monclient(hunting): _negotiate 2024-03-25T14:25:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.556+0000 7f05ad0f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:49.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.556+0000 7f05ad0f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.556+0000 7f05ad0f7640 10 monclient(hunting): my global_id is 4469 2024-03-25T14:25:49.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.556+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a8142520 con 0x7f05a80a98f0 2024-03-25T14:25:49.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.564+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900369878 0 0) 0x7f05a80b28a0 con 0x7f05a81430c0 2024-03-25T14:25:49.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.564+0000 7f05ad0f7640 10 monclient(hunting): _negotiate 2024-03-25T14:25:49.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.564+0000 7f05ad0f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:49.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.564+0000 7f05ad0f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:49.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.564+0000 7f05ad0f7640 10 monclient(hunting): my global_id is 4492 2024-03-25T14:25:49.569 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.564+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a80b2a30 con 0x7f05a81430c0 2024-03-25T14:25:49.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4053100148 0 0) 0x7f0594003520 con 0x7f05a813f710 2024-03-25T14:25:49.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 >> v1:172.21.15.63:6790/0 conn(0x7f05a80a98f0 legacy=0x7f05a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:49.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 >> v1:172.21.15.153:6789/0 conn(0x7f05a81430c0 legacy=0x7f05a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 10 monclient: _finish_hunting 0 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 1 monclient: found mon.noname-a 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a80b44c0 con 0x7f05a813f710 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 10 monclient: _finish_auth 0 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 10 monclient: _check_auth_tickets 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0598001690 con 0x7f05a813f710 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05ad0f7640 10 monclient: dump: 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:49.586 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05b035d640 5 monclient: authenticate success, global_id 4491 2024-03-25T14:25:49.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.581+0000 7f05b035d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05980017f0 con 0x7f05a813f710 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05ad0f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05ad0f7640 1 -- 172.21.15.63:0/1847710984 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0598001c00 con 0x7f05a813f710 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05b035d640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05b035d640 4 set_mon_vals no callback set 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05ad0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05ad0f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.596+0000 7f05ad0f7640 10 monclient: dump: 2024-03-25T14:25:49.600 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:49.601 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:49.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.597+0000 7f05b035d640 10 monclient: shutdown 2024-03-25T14:25:49.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.597+0000 7f05b035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:49.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.597+0000 7f05b035d640 1 -- 172.21.15.63:0/1847710984 >> v1:172.21.15.63:6789/0 conn(0x7f05a813f710 legacy=0x7f05a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:49.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.597+0000 7f05b035d640 1 -- 172.21.15.63:0/1847710984 shutdown_connections 2024-03-25T14:25:49.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.597+0000 7f05b035d640 1 -- 172.21.15.63:0/1847710984 >> 172.21.15.63:0/1847710984 conn(0x7f05a80b7ea0 msgr2=0x7f05a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:49.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.597+0000 7f05b035d640 1 -- 172.21.15.63:0/1847710984 shutdown_connections 2024-03-25T14:25:49.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.597+0000 7f05b035d640 1 -- 172.21.15.63:0/1847710984 wait complete. 2024-03-25T14:25:49.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.598+0000 7f05b035d640 10 monclient: build_initial_monmap 2024-03-25T14:25:49.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.598+0000 7f05b035d640 10 monclient: monmap: 2024-03-25T14:25:49.604 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:49.604 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:49.604 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 1 librados: starting msgr at 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 1 librados: starting objecter 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 1 Processor -- start 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 1 -- start start 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 1 librados: setting wanted keys 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 1 librados: calling monclient init 2024-03-25T14:25:49.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 10 monclient: init 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 10 monclient(hunting): picked mon.noname-c con 0x7f05a80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 10 monclient(hunting): picked mon.noname-b con 0x7f05a813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 10 monclient(hunting): picked mon.noname-a con 0x7f05a81430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.599+0000 7f05b035d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a81dbaa0 con 0x7f05a81430c0 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.600+0000 7f05b035d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a806f550 con 0x7f05a813f710 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.600+0000 7f05b035d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a8070b90 con 0x7f05a80a98f0 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.600+0000 7f05b035d640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:49.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.600+0000 7f05b035d640 10 monclient(hunting): authenticate will time out at 1712.744586s 2024-03-25T14:25:49.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.610+0000 7f05ae8fa640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f05a81430c0 0x7f05a81e6630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:57754/0 (socket says 172.21.15.63:57754) 2024-03-25T14:25:49.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.610+0000 7f05ae8fa640 1 -- 172.21.15.63:0/1739778238 learned_addr learned my addr 172.21.15.63:0/1739778238 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:49.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.613+0000 7f059effd640 10 client.?.objecter ms_handle_connect 0x7f05a81430c0 2024-03-25T14:25:49.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.613+0000 7f059effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:49.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.614+0000 7f059effd640 10 client.?.objecter ms_handle_connect 0x7f05a80a98f0 2024-03-25T14:25:49.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.614+0000 7f059effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:49.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.621+0000 7f059effd640 10 client.?.objecter ms_handle_connect 0x7f05a813f710 2024-03-25T14:25:49.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.621+0000 7f059effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:49.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.625+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196642070 0 0) 0x7f05a81dbaa0 con 0x7f05a81430c0 2024-03-25T14:25:49.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.625+0000 7f059effd640 10 monclient(hunting): _negotiate 2024-03-25T14:25:49.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.625+0000 7f059effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:49.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.625+0000 7f059effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:49.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.625+0000 7f059effd640 10 monclient(hunting): my global_id is 4494 2024-03-25T14:25:49.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.625+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c004eb0 con 0x7f05a81430c0 2024-03-25T14:25:49.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.630+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744287245 0 0) 0x7f05a8070b90 con 0x7f05a80a98f0 2024-03-25T14:25:49.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.630+0000 7f059effd640 10 monclient(hunting): _negotiate 2024-03-25T14:25:49.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.630+0000 7f059effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:49.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.630+0000 7f059effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:49.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.630+0000 7f059effd640 10 monclient(hunting): my global_id is 4472 2024-03-25T14:25:49.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.630+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a81dbaa0 con 0x7f05a80a98f0 2024-03-25T14:25:49.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.631+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658148082 0 0) 0x7f05a806f550 con 0x7f05a813f710 2024-03-25T14:25:49.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.631+0000 7f059effd640 10 monclient(hunting): _negotiate 2024-03-25T14:25:49.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.631+0000 7f059effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:49.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.631+0000 7f059effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:49.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.631+0000 7f059effd640 10 monclient(hunting): my global_id is 4495 2024-03-25T14:25:49.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.631+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a8070b90 con 0x7f05a813f710 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.635+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3236675619 0 0) 0x7f058c004eb0 con 0x7f05a81430c0 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.635+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05a806f550 con 0x7f05a81430c0 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.635+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f05a0001c70 con 0x7f05a81430c0 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.635+0000 7f059effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.635+0000 7f059effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.635+0000 7f059effd640 10 monclient(hunting): dump: 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:49.639 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:49.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.635+0000 7f059effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:49.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.649+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4141025259 0 0) 0x7f05a8070b90 con 0x7f05a813f710 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.650+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f058c004eb0 con 0x7f05a813f710 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.650+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0598001b00 con 0x7f05a813f710 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.650+0000 7f059effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.650+0000 7f059effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.650+0000 7f059effd640 10 monclient(hunting): dump: 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:49.654 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:49.655 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:49.655 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:49.655 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2354968132 0 0) 0x7f05a806f550 con 0x7f05a81430c0 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 >> v1:172.21.15.63:6790/0 conn(0x7f05a80a98f0 legacy=0x7f05a806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 >> v1:172.21.15.153:6789/0 conn(0x7f05a813f710 legacy=0x7f05a81dbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 10 monclient: _finish_hunting 0 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 1 monclient: found mon.a 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8070d20 con 0x7f05a81430c0 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 10 monclient: _finish_auth 0 2024-03-25T14:25:49.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 10 monclient: _check_auth_tickets 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f059effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f05b035d640 5 monclient: authenticate success, global_id 4494 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f05b035d640 10 monclient: _renew_subs 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f05b035d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.651+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a81dc6d0 con 0x7f05a81430c0 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.652+0000 7f05b035d640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.652+0000 7f05b035d640 10 monclient: _renew_subs 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.652+0000 7f05b035d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.652+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a806e280 con 0x7f05a81430c0 2024-03-25T14:25:49.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.652+0000 7f05b035d640 1 librados: init done 2024-03-25T14:25:49.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.653+0000 7f05b035d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:49.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.653+0000 7f05b035d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:49.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.653+0000 7f05b035d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:49.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.653+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a81dca20 con 0x7f05a81430c0 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.968+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05a0003990 con 0x7f05a81430c0 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.968+0000 7f059effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.968+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f05a0003da0 con 0x7f05a81430c0 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.968+0000 7f059effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.968+0000 7f059effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.968+0000 7f059effd640 10 monclient: dump: 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:49.973 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.968+0000 7f05ac8f6640 4 set_mon_vals no callback set 2024-03-25T14:25:49.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.969+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f05a0003fc0 con 0x7f05a81430c0 2024-03-25T14:25:49.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.970+0000 7f05ae0f9640 10 monclient: get_auth_request con 0x7f058c06df90 auth_method 0 2024-03-25T14:25:49.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.970+0000 7f059effd640 10 client.4494.objecter ms_handle_connect 0x7f058c06df90 2024-03-25T14:25:49.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f05a0054a90 con 0x7f05a81430c0 2024-03-25T14:25:49.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 10 client.4494.objecter ms_dispatch 0x7f05a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:49.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:49.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:49.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T14:25:49.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05a00505a0 con 0x7f05a81430c0 2024-03-25T14:25:49.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:49.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:49.975+0000 7f059effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:50.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.228+0000 7f05b035d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:50.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.228+0000 7f05b035d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:50.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.228+0000 7f05b035d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:50.233 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.228+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05a8074cc0 con 0x7f05a81430c0 2024-03-25T14:25:50.281 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.277+0000 7f059effd640 1 -- 172.21.15.63:0/1739778238 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f05a00524d0 con 0x7f05a81430c0 2024-03-25T14:25:50.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.277+0000 7f059effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:50.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.277+0000 7f059effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:50.282 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:50.282 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 10 librados: watch_flush enter 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 10 librados: watch_flush exit 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 10 client.4494.objecter successfully canceled tick 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 >> v1:172.21.15.153:6800/1688255552 conn(0x7f058c06df90 legacy=0x7f058c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 10 monclient: shutdown 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 >> v1:172.21.15.63:6789/0 conn(0x7f05a81430c0 legacy=0x7f05a81e6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:50.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 shutdown_connections 2024-03-25T14:25:50.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.279+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 >> 172.21.15.63:0/1739778238 conn(0x7f05a80b7ea0 msgr2=0x7f05a814e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:50.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.280+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 shutdown_connections 2024-03-25T14:25:50.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.280+0000 7f05b035d640 1 -- 172.21.15.63:0/1739778238 wait complete. 2024-03-25T14:25:50.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:50.280+0000 7f05b035d640 1 librados: shutdown 2024-03-25T14:25:50.296 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:25:57.103 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.214+0000 7fd2751c6640 10 monclient: get_monmap_and_config 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.214+0000 7fd2751c6640 10 monclient: build_initial_monmap 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.214+0000 7fd2751c6640 10 monclient: monmap: 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:25:57.215907+0000 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:25:57.215907+0000 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:57.219 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:57.220 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:25:57.220 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:25:57.220 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.215+0000 7fd2751c6640 1 Processor -- start 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.215+0000 7fd2751c6640 1 -- start start 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.215+0000 7fd2751c6640 10 monclient: init 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2700a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 10 monclient(hunting): picked mon.noname-b con 0x7fd27013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2701430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd270142520 con 0x7fd2701430c0 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2700b28a0 con 0x7fd2700a98f0 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2700b2a30 con 0x7fd27013f710 2024-03-25T14:25:57.221 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:57.222 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.216+0000 7fd2751c6640 10 monclient(hunting): authenticate will time out at 1720.360949s 2024-03-25T14:25:57.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.222+0000 7fd26ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd2700a98f0 0x7fd2700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:43276/0 (socket says 172.21.15.63:43276) 2024-03-25T14:25:57.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.222+0000 7fd26ed9d640 1 -- 172.21.15.63:0/3442433684 learned_addr learned my addr 172.21.15.63:0/3442433684 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:57.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.228+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858344591 0 0) 0x7fd2700b28a0 con 0x7fd2700a98f0 2024-03-25T14:25:57.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.228+0000 7fd26dd9b640 10 monclient(hunting): _negotiate 2024-03-25T14:25:57.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.228+0000 7fd26dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:57.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.228+0000 7fd26dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:57.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.228+0000 7fd26dd9b640 10 monclient(hunting): my global_id is 4498 2024-03-25T14:25:57.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.228+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd258003520 con 0x7fd2700a98f0 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47787466 0 0) 0x7fd258003520 con 0x7fd2700a98f0 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 >> v1:172.21.15.63:6790/0 conn(0x7fd27013f710 legacy=0x7fd2701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 >> v1:172.21.15.63:6789/0 conn(0x7fd2701430c0 legacy=0x7fd27014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 10 monclient: _finish_hunting 0 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 1 monclient: found mon.noname-c 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2700b44c0 con 0x7fd2700a98f0 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 10 monclient: _finish_auth 0 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 10 monclient: _check_auth_tickets 2024-03-25T14:25:57.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd260001690 con 0x7fd2700a98f0 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd26dd9b640 10 monclient: dump: 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:57.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd2751c6640 5 monclient: authenticate success, global_id 4498 2024-03-25T14:25:57.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.237+0000 7fd2751c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd260001860 con 0x7fd2700a98f0 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd26dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd26dd9b640 1 -- 172.21.15.63:0/3442433684 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd260001cb0 con 0x7fd2700a98f0 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd26dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd26dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd26dd9b640 10 monclient: dump: 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:57.255 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd2751c6640 10 monclient: get_monmap_and_config success 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.249+0000 7fd2751c6640 4 set_mon_vals no callback set 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.250+0000 7fd2751c6640 10 monclient: shutdown 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.250+0000 7fd2751c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.250+0000 7fd2751c6640 1 -- 172.21.15.63:0/3442433684 >> v1:172.21.15.153:6789/0 conn(0x7fd2700a98f0 legacy=0x7fd2700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.250+0000 7fd2751c6640 1 -- 172.21.15.63:0/3442433684 shutdown_connections 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.250+0000 7fd2751c6640 1 -- 172.21.15.63:0/3442433684 >> 172.21.15.63:0/3442433684 conn(0x7fd2700b7ea0 msgr2=0x7fd2700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.251+0000 7fd2751c6640 1 -- 172.21.15.63:0/3442433684 shutdown_connections 2024-03-25T14:25:57.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.251+0000 7fd2751c6640 1 -- 172.21.15.63:0/3442433684 wait complete. 2024-03-25T14:25:57.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.252+0000 7fd2751c6640 10 monclient: build_initial_monmap 2024-03-25T14:25:57.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.252+0000 7fd2751c6640 10 monclient: monmap: 2024-03-25T14:25:57.258 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.252+0000 7fd2751c6640 1 librados: starting msgr at 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.252+0000 7fd2751c6640 1 librados: starting objecter 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.252+0000 7fd2751c6640 1 Processor -- start 2024-03-25T14:25:57.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 1 -- start start 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 1 librados: setting wanted keys 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 1 librados: calling monclient init 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient: init 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient: _reopen_session rank -1 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2700a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient(hunting): picked mon.noname-b con 0x7fd27013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2701430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd270070170 con 0x7fd2700a98f0 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd270071280 con 0x7fd27013f710 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd270071410 con 0x7fd2701430c0 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient(hunting): _renew_subs 2024-03-25T14:25:57.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.253+0000 7fd2751c6640 10 monclient(hunting): authenticate will time out at 1720.398295s 2024-03-25T14:25:57.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.262+0000 7fd26e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd27013f710 0x7fd27006e210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:43326/0 (socket says 172.21.15.63:43326) 2024-03-25T14:25:57.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.262+0000 7fd26e59c640 1 -- 172.21.15.63:0/1957289419 learned_addr learned my addr 172.21.15.63:0/1957289419 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:25:57.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.262+0000 7fd2577fe640 10 client.?.objecter ms_handle_connect 0x7fd27013f710 2024-03-25T14:25:57.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.262+0000 7fd2577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:57.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.264+0000 7fd2577fe640 10 client.?.objecter ms_handle_connect 0x7fd2701430c0 2024-03-25T14:25:57.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.264+0000 7fd2577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.334+0000 7fd2577fe640 10 client.?.objecter ms_handle_connect 0x7fd2700a98f0 2024-03-25T14:25:57.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.334+0000 7fd2577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:25:57.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.350+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781820934 0 0) 0x7fd270070170 con 0x7fd2700a98f0 2024-03-25T14:25:57.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.350+0000 7fd2577fe640 10 monclient(hunting): _negotiate 2024-03-25T14:25:57.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.350+0000 7fd2577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:57.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.350+0000 7fd2577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:57.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.350+0000 7fd2577fe640 10 monclient(hunting): my global_id is 4497 2024-03-25T14:25:57.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.350+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25c004eb0 con 0x7fd2700a98f0 2024-03-25T14:25:57.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880681260 0 0) 0x7fd270071280 con 0x7fd27013f710 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): _negotiate 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd270070170 con 0x7fd27013f710 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032113246 0 0) 0x7fd25c004eb0 con 0x7fd2700a98f0 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd270071280 con 0x7fd2700a98f0 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd260001b70 con 0x7fd2700a98f0 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): dump: 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:57.388 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:57.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.383+0000 7fd2577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:25:57.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.391+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936036724 0 0) 0x7fd270071410 con 0x7fd2701430c0 2024-03-25T14:25:57.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.391+0000 7fd2577fe640 10 monclient(hunting): _negotiate 2024-03-25T14:25:57.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.391+0000 7fd2577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:25:57.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.391+0000 7fd2577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:25:57.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.391+0000 7fd2577fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T14:25:57.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.391+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25c004eb0 con 0x7fd2701430c0 2024-03-25T14:25:57.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.397+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1420459026 0 0) 0x7fd270070170 con 0x7fd27013f710 2024-03-25T14:25:57.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.397+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd270071410 con 0x7fd27013f710 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.397+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd264004000 con 0x7fd27013f710 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.397+0000 7fd2577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.397+0000 7fd2577fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.397+0000 7fd2577fe640 10 monclient(hunting): dump: 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:57.402 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:57.403 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1271475176 0 0) 0x7fd270071280 con 0x7fd2700a98f0 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 >> v1:172.21.15.63:6790/0 conn(0x7fd2701430c0 legacy=0x7fd27006e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 >> v1:172.21.15.153:6789/0 conn(0x7fd27013f710 legacy=0x7fd27006e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 10 monclient: _finish_hunting 0 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 1 monclient: found mon.a 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd270071de0 con 0x7fd2700a98f0 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 10 monclient: _finish_auth 0 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 10 monclient: _check_auth_tickets 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 5 monclient: authenticate success, global_id 4497 2024-03-25T14:25:57.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 10 monclient: _renew_subs 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd270072090 con 0x7fd2700a98f0 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 10 monclient: _renew_subs 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2701e73b0 con 0x7fd2700a98f0 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.406+0000 7fd2751c6640 1 librados: init done 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.408+0000 7fd2751c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.408+0000 7fd2751c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.408+0000 7fd2751c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:57.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.408+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2700627b0 con 0x7fd2700a98f0 2024-03-25T14:25:57.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.723+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd260001010 con 0x7fd2700a98f0 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.723+0000 7fd2577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.723+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd260004bf0 con 0x7fd2700a98f0 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.723+0000 7fd2577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.723+0000 7fd2577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.723+0000 7fd2577fe640 10 monclient: dump: 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:25:57.728 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:25:57.729 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:25:57.729 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:25:57.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.723+0000 7fd26d59a640 4 set_mon_vals no callback set 2024-03-25T14:25:57.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.724+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd26001abc0 con 0x7fd2700a98f0 2024-03-25T14:25:57.729 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.725+0000 7fd26e59c640 10 monclient: get_auth_request con 0x7fd25c06dec0 auth_method 0 2024-03-25T14:25:57.730 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.725+0000 7fd2577fe640 10 client.4497.objecter ms_handle_connect 0x7fd25c06dec0 2024-03-25T14:25:57.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fd2600548e0 con 0x7fd2700a98f0 2024-03-25T14:25:57.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 10 client.4497.objecter ms_dispatch 0x7fd270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:25:57.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:25:57.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:25:57.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T14:25:57.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd260050220 con 0x7fd2700a98f0 2024-03-25T14:25:57.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:25:57.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.730+0000 7fd2577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:25:57.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.987+0000 7fd2751c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:57.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.987+0000 7fd2751c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:57.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.987+0000 7fd2751c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:25:57.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:57.987+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2701419e0 con 0x7fd2700a98f0 2024-03-25T14:25:58.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.029+0000 7fd2577fe640 1 -- 172.21.15.63:0/1957289419 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fd260052320 con 0x7fd2700a98f0 2024-03-25T14:25:58.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.029+0000 7fd2577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:25:58.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.029+0000 7fd2577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:25:58.033 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:25:58.034 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:25:58.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 10 librados: watch_flush enter 2024-03-25T14:25:58.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 10 librados: watch_flush exit 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 10 client.4497.objecter successfully canceled tick 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd25c06dec0 legacy=0x7fd25c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 10 monclient: shutdown 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 >> v1:172.21.15.63:6789/0 conn(0x7fd2700a98f0 legacy=0x7fd270070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 shutdown_connections 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 >> 172.21.15.63:0/1957289419 conn(0x7fd2700b7ea0 msgr2=0x7fd270074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 shutdown_connections 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 1 -- 172.21.15.63:0/1957289419 wait complete. 2024-03-25T14:25:58.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:25:58.031+0000 7fd2751c6640 1 librados: shutdown 2024-03-25T14:25:58.048 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:04.855 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:04.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.963+0000 7ff056ded640 10 monclient: get_monmap_and_config 2024-03-25T14:26:04.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.963+0000 7ff056ded640 10 monclient: build_initial_monmap 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.963+0000 7ff056ded640 10 monclient: monmap: 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:04.964467+0000 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:04.964467+0000 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:04.968 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.964+0000 7ff056ded640 1 Processor -- start 2024-03-25T14:26:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.964+0000 7ff056ded640 1 -- start start 2024-03-25T14:26:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.964+0000 7ff056ded640 10 monclient: init 2024-03-25T14:26:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:26:04.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 10 monclient(hunting): picked mon.noname-a con 0x7ff0500b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 10 monclient(hunting): picked mon.noname-b con 0x7ff0500b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 10 monclient(hunting): picked mon.noname-c con 0x7ff05014be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500af940 con 0x7ff0500b4d60 2024-03-25T14:26:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500b0a70 con 0x7ff05014be90 2024-03-25T14:26:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500b0d50 con 0x7ff0500b1730 2024-03-25T14:26:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:04.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.965+0000 7ff056ded640 10 monclient(hunting): authenticate will time out at 1728.109751s 2024-03-25T14:26:04.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.973+0000 7ff05538a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff05014be90 0x7ff05014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:59510/0 (socket says 172.21.15.63:59510) 2024-03-25T14:26:04.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.973+0000 7ff05538a640 1 -- 172.21.15.63:0/1056916895 learned_addr learned my addr 172.21.15.63:0/1056916895 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:04.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.988+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061715451 0 0) 0x7ff0500b0a70 con 0x7ff05014be90 2024-03-25T14:26:04.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.988+0000 7ff04f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:04.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.988+0000 7ff04f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:04.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.988+0000 7ff04f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:04.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.988+0000 7ff04f7fe640 10 monclient(hunting): my global_id is 4504 2024-03-25T14:26:04.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.988+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff040003520 con 0x7ff05014be90 2024-03-25T14:26:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.991+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002239631 0 0) 0x7ff0500b0d50 con 0x7ff0500b1730 2024-03-25T14:26:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.991+0000 7ff04f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.991+0000 7ff04f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.991+0000 7ff04f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.991+0000 7ff04f7fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T14:26:04.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:04.991+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0500b0a70 con 0x7ff0500b1730 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 636245519 0 0) 0x7ff040003520 con 0x7ff05014be90 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 >> v1:172.21.15.63:6790/0 conn(0x7ff0500b1730 legacy=0x7ff0500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 >> v1:172.21.15.63:6789/0 conn(0x7ff0500b4d60 legacy=0x7ff05013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 1 monclient: found mon.noname-c 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0500b1030 con 0x7ff05014be90 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff048001a10 con 0x7ff05014be90 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff056ded640 5 monclient: authenticate success, global_id 4504 2024-03-25T14:26:05.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff056ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff04f7fe640 10 monclient: dump: 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:05.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.009+0000 7ff056ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:05.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff048001b70 con 0x7ff05014be90 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff04f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff04f7fe640 1 -- 172.21.15.63:0/1056916895 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff048001f80 con 0x7ff05014be90 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff04f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff04f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff04f7fe640 10 monclient: dump: 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:05.034 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:05.035 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:05.035 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:05.035 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:05.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff056ded640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:05.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.029+0000 7ff056ded640 4 set_mon_vals no callback set 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.030+0000 7ff056ded640 10 monclient: shutdown 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.030+0000 7ff056ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.030+0000 7ff056ded640 1 -- 172.21.15.63:0/1056916895 >> v1:172.21.15.153:6789/0 conn(0x7ff05014be90 legacy=0x7ff05014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.030+0000 7ff056ded640 1 -- 172.21.15.63:0/1056916895 shutdown_connections 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.030+0000 7ff056ded640 1 -- 172.21.15.63:0/1056916895 >> 172.21.15.63:0/1056916895 conn(0x7ff0500617a0 msgr2=0x7ff050061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.030+0000 7ff056ded640 1 -- 172.21.15.63:0/1056916895 shutdown_connections 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.030+0000 7ff056ded640 1 -- 172.21.15.63:0/1056916895 wait complete. 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 10 monclient: build_initial_monmap 2024-03-25T14:26:05.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 10 monclient: monmap: 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:05.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 1 librados: starting msgr at 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 1 librados: starting objecter 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 1 Processor -- start 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 1 -- start start 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 1 librados: setting wanted keys 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 1 librados: calling monclient init 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.031+0000 7ff056ded640 10 monclient: init 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 10 monclient(hunting): picked mon.noname-c con 0x7ff0500b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 10 monclient(hunting): picked mon.noname-a con 0x7ff0500b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 10 monclient(hunting): picked mon.noname-b con 0x7ff05014be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:05.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff050135ee0 con 0x7ff0500b4d60 2024-03-25T14:26:05.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff050135b00 con 0x7ff05014be90 2024-03-25T14:26:05.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff050135c90 con 0x7ff0500b1730 2024-03-25T14:26:05.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:05.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.032+0000 7ff056ded640 10 monclient(hunting): authenticate will time out at 1728.177439s 2024-03-25T14:26:05.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.043+0000 7ff05538a640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff05014be90 0x7ff0501d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:59570/0 (socket says 172.21.15.63:59570) 2024-03-25T14:26:05.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.043+0000 7ff05538a640 1 -- 172.21.15.63:0/1006393440 learned_addr learned my addr 172.21.15.63:0/1006393440 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:05.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.044+0000 7ff04d7fa640 10 client.?.objecter ms_handle_connect 0x7ff05014be90 2024-03-25T14:26:05.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.044+0000 7ff04d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:05.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.047+0000 7ff04d7fa640 10 client.?.objecter ms_handle_connect 0x7ff0500b1730 2024-03-25T14:26:05.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.047+0000 7ff04d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:05.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.052+0000 7ff04d7fa640 10 client.?.objecter ms_handle_connect 0x7ff0500b4d60 2024-03-25T14:26:05.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.052+0000 7ff04d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:05.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.056+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085336399 0 0) 0x7ff050135b00 con 0x7ff05014be90 2024-03-25T14:26:05.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.057+0000 7ff04d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:05.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.057+0000 7ff04d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:05.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.057+0000 7ff04d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:05.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.057+0000 7ff04d7fa640 10 monclient(hunting): my global_id is 4507 2024-03-25T14:26:05.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.057+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff034004eb0 con 0x7ff05014be90 2024-03-25T14:26:05.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.059+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087237394 0 0) 0x7ff050135c90 con 0x7ff0500b1730 2024-03-25T14:26:05.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.059+0000 7ff04d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:05.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.059+0000 7ff04d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:05.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.059+0000 7ff04d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:05.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.059+0000 7ff04d7fa640 10 monclient(hunting): my global_id is 4481 2024-03-25T14:26:05.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.059+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff050135b00 con 0x7ff0500b1730 2024-03-25T14:26:05.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.062+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134245861 0 0) 0x7ff050135ee0 con 0x7ff0500b4d60 2024-03-25T14:26:05.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.062+0000 7ff04d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:05.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.062+0000 7ff04d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:05.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.062+0000 7ff04d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:05.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.062+0000 7ff04d7fa640 10 monclient(hunting): my global_id is 4503 2024-03-25T14:26:05.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.062+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff050135c90 con 0x7ff0500b4d60 2024-03-25T14:26:05.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.069+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736929122 0 0) 0x7ff034004eb0 con 0x7ff05014be90 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.069+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff050135ee0 con 0x7ff05014be90 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.069+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff048001f40 con 0x7ff05014be90 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.069+0000 7ff04d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.069+0000 7ff04d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.069+0000 7ff04d7fa640 10 monclient(hunting): dump: 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:05.074 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:05.075 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:05.075 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:05.075 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:05.075 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.069+0000 7ff04d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.077+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764433916 0 0) 0x7ff050135c90 con 0x7ff0500b4d60 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.077+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff034004eb0 con 0x7ff0500b4d60 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.077+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff03c001930 con 0x7ff0500b4d60 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.077+0000 7ff04d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.077+0000 7ff04d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.077+0000 7ff04d7fa640 10 monclient(hunting): dump: 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.081 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.078+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270901469 0 0) 0x7ff050135b00 con 0x7ff0500b1730 2024-03-25T14:26:05.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.078+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff050135c90 con 0x7ff0500b1730 2024-03-25T14:26:05.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.084+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff044001920 con 0x7ff0500b1730 2024-03-25T14:26:05.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.084+0000 7ff04d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:05.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.084+0000 7ff04d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:26:05.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.084+0000 7ff04d7fa640 10 monclient(hunting): dump: 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:05.089 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:05.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.095+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 918485902 0 0) 0x7ff050135ee0 con 0x7ff05014be90 2024-03-25T14:26:05.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.095+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 >> v1:172.21.15.63:6790/0 conn(0x7ff0500b1730 legacy=0x7ff05012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:05.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 >> v1:172.21.15.63:6789/0 conn(0x7ff0500b4d60 legacy=0x7ff0501d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 1 monclient: found mon.b 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050136960 con 0x7ff05014be90 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff04d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 5 monclient: authenticate success, global_id 4507 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 10 monclient: _renew_subs 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff050130470 con 0x7ff05014be90 2024-03-25T14:26:05.101 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:05.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 10 monclient: _renew_subs 2024-03-25T14:26:05.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:05.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05013e690 con 0x7ff05014be90 2024-03-25T14:26:05.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.096+0000 7ff056ded640 1 librados: init done 2024-03-25T14:26:05.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.097+0000 7ff056ded640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:05.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.097+0000 7ff056ded640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:05.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.097+0000 7ff056ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:05.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.097+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff050136b50 con 0x7ff05014be90 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.402+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff048004e10 con 0x7ff05014be90 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.402+0000 7ff04d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.402+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff048004f80 con 0x7ff05014be90 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.402+0000 7ff04d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.402+0000 7ff04effd640 4 set_mon_vals no callback set 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.402+0000 7ff04d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.402+0000 7ff04d7fa640 10 monclient: dump: 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.407 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:05.408 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:05.408 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:05.408 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:05.408 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:05.408 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:05.408 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:05.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.403+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7ff04801af50 con 0x7ff05014be90 2024-03-25T14:26:05.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.404+0000 7ff054b89640 10 monclient: get_auth_request con 0x7ff03406e040 auth_method 0 2024-03-25T14:26:05.409 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.404+0000 7ff04d7fa640 10 client.4507.objecter ms_handle_connect 0x7ff03406e040 2024-03-25T14:26:05.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.408+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7ff048054990 con 0x7ff05014be90 2024-03-25T14:26:05.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.408+0000 7ff04d7fa640 10 client.4507.objecter ms_dispatch 0x7ff050000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:05.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.408+0000 7ff04d7fa640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:05.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.408+0000 7ff04d7fa640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:05.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.408+0000 7ff04d7fa640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T14:26:05.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.409+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0480504a0 con 0x7ff05014be90 2024-03-25T14:26:05.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.409+0000 7ff04d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:05.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.409+0000 7ff04d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:05.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.637+0000 7ff056ded640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:05.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.637+0000 7ff056ded640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:05.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.637+0000 7ff056ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:05.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.637+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff050136090 con 0x7ff05014be90 2024-03-25T14:26:05.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.818+0000 7ff04d7fa640 1 -- 172.21.15.63:0/1006393440 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7ff050136090 con 0x7ff05014be90 2024-03-25T14:26:05.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.818+0000 7ff04d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:05.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.818+0000 7ff04d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:05.823 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:05.823 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 10 librados: watch_flush enter 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 10 librados: watch_flush exit 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 10 client.4507.objecter successfully canceled tick 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 >> v1:172.21.15.153:6800/1688255552 conn(0x7ff03406e040 legacy=0x7ff034070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 10 monclient: shutdown 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 >> v1:172.21.15.153:6789/0 conn(0x7ff05014be90 legacy=0x7ff0501d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 shutdown_connections 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 >> 172.21.15.63:0/1006393440 conn(0x7ff0500617a0 msgr2=0x7ff0500b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 shutdown_connections 2024-03-25T14:26:05.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 1 -- 172.21.15.63:0/1006393440 wait complete. 2024-03-25T14:26:05.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:05.821+0000 7ff056ded640 1 librados: shutdown 2024-03-25T14:26:05.837 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:07.056 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:45.568 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.114 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:45.626 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.149 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:06:45.662 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.164 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:45.624 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.168 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:45.680 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.173 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:45.633 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.178 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:45.690 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.202 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:45.662 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:07.210 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:06:45.723 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:12.643 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:12.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.754+0000 7fa12797b640 10 monclient: get_monmap_and_config 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.754+0000 7fa12797b640 10 monclient: build_initial_monmap 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.754+0000 7fa12797b640 10 monclient: monmap: 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:12.755691+0000 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:12.755691+0000 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:12.759 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:12.760 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:12.760 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:12.760 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:12.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.755+0000 7fa12797b640 1 Processor -- start 2024-03-25T14:26:12.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.755+0000 7fa12797b640 1 -- start start 2024-03-25T14:26:12.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.755+0000 7fa12797b640 10 monclient: init 2024-03-25T14:26:12.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:12.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:26:12.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 10 monclient(hunting): picked mon.noname-b con 0x7fa1200a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:12.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 10 monclient(hunting): picked mon.noname-c con 0x7fa12013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:12.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 10 monclient(hunting): picked mon.noname-a con 0x7fa1201430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:12.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa120142520 con 0x7fa1201430c0 2024-03-25T14:26:12.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1200b28a0 con 0x7fa12013f710 2024-03-25T14:26:12.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1200b2a30 con 0x7fa1200a98f0 2024-03-25T14:26:12.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:12.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.756+0000 7fa12797b640 10 monclient(hunting): authenticate will time out at 1735.901041s 2024-03-25T14:26:12.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.764+0000 7fa124f16640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa12013f710 0x7fa1201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:39986/0 (socket says 172.21.15.63:39986) 2024-03-25T14:26:12.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.764+0000 7fa124f16640 1 -- 172.21.15.63:0/2782460891 learned_addr learned my addr 172.21.15.63:0/2782460891 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:12.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.773+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308350889 0 0) 0x7fa1200b28a0 con 0x7fa12013f710 2024-03-25T14:26:12.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.773+0000 7fa117fff640 10 monclient(hunting): _negotiate 2024-03-25T14:26:12.777 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.773+0000 7fa117fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:12.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.773+0000 7fa117fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:12.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.773+0000 7fa117fff640 10 monclient(hunting): my global_id is 4510 2024-03-25T14:26:12.778 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.773+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa10c003520 con 0x7fa12013f710 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1140534028 0 0) 0x7fa10c003520 con 0x7fa12013f710 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 >> v1:172.21.15.63:6790/0 conn(0x7fa1200a98f0 legacy=0x7fa1200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 >> v1:172.21.15.63:6789/0 conn(0x7fa1201430c0 legacy=0x7fa12014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 10 monclient: _finish_hunting 0 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 1 monclient: found mon.noname-c 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:26:12.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1200b44c0 con 0x7fa12013f710 2024-03-25T14:26:12.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 10 monclient: _finish_auth 0 2024-03-25T14:26:12.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 10 monclient: _check_auth_tickets 2024-03-25T14:26:12.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa117fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:12.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa12797b640 5 monclient: authenticate success, global_id 4510 2024-03-25T14:26:12.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.786+0000 7fa12797b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa110001690 con 0x7fa12013f710 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 10 monclient: dump: 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:12.795 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa110001920 con 0x7fa12013f710 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa12797b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa12797b640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:12.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 1 -- 172.21.15.63:0/2782460891 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa110001c30 con 0x7fa12013f710 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa12797b640 4 set_mon_vals no callback set 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.790+0000 7fa117fff640 10 monclient: dump: 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:12.797 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:12.798 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:12.798 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:12.798 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:12.798 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:12.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.791+0000 7fa12797b640 10 monclient: shutdown 2024-03-25T14:26:12.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.791+0000 7fa12797b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.791+0000 7fa12797b640 1 -- 172.21.15.63:0/2782460891 >> v1:172.21.15.153:6789/0 conn(0x7fa12013f710 legacy=0x7fa1201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.791+0000 7fa12797b640 1 -- 172.21.15.63:0/2782460891 shutdown_connections 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.791+0000 7fa12797b640 1 -- 172.21.15.63:0/2782460891 >> 172.21.15.63:0/2782460891 conn(0x7fa1200b7ea0 msgr2=0x7fa1200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.791+0000 7fa12797b640 1 -- 172.21.15.63:0/2782460891 shutdown_connections 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.791+0000 7fa12797b640 1 -- 172.21.15.63:0/2782460891 wait complete. 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.792+0000 7fa12797b640 10 monclient: build_initial_monmap 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.792+0000 7fa12797b640 10 monclient: monmap: 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:12.801 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.792+0000 7fa12797b640 1 librados: starting msgr at 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.792+0000 7fa12797b640 1 librados: starting objecter 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.793+0000 7fa12797b640 1 Processor -- start 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.793+0000 7fa12797b640 1 -- start start 2024-03-25T14:26:12.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.793+0000 7fa12797b640 1 librados: setting wanted keys 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.793+0000 7fa12797b640 1 librados: calling monclient init 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.793+0000 7fa12797b640 10 monclient: init 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.793+0000 7fa12797b640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 10 monclient(hunting): picked mon.noname-a con 0x7fa1200a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 10 monclient(hunting): picked mon.noname-c con 0x7fa12013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 10 monclient(hunting): picked mon.noname-b con 0x7fa1201430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa120139990 con 0x7fa1200a98f0 2024-03-25T14:26:12.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1201d7880 con 0x7fa1201430c0 2024-03-25T14:26:12.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1201d84b0 con 0x7fa12013f710 2024-03-25T14:26:12.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:12.804 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.794+0000 7fa12797b640 10 monclient(hunting): authenticate will time out at 1735.938630s 2024-03-25T14:26:12.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.802+0000 7fa125f18640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa1201430c0 0x7fa120138540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:40038/0 (socket says 172.21.15.63:40038) 2024-03-25T14:26:12.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.802+0000 7fa125f18640 1 -- 172.21.15.63:0/1562242143 learned_addr learned my addr 172.21.15.63:0/1562242143 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:12.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.803+0000 7fa115ffb640 10 client.?.objecter ms_handle_connect 0x7fa1201430c0 2024-03-25T14:26:12.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.803+0000 7fa115ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:12.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.807+0000 7fa115ffb640 10 client.?.objecter ms_handle_connect 0x7fa12013f710 2024-03-25T14:26:12.811 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.807+0000 7fa115ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:12.814 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.811+0000 7fa115ffb640 10 client.?.objecter ms_handle_connect 0x7fa1200a98f0 2024-03-25T14:26:12.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.811+0000 7fa115ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:12.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.812+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280486039 0 0) 0x7fa1201d7880 con 0x7fa1201430c0 2024-03-25T14:26:12.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.812+0000 7fa115ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:26:12.816 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.812+0000 7fa115ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:12.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.812+0000 7fa115ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:12.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.812+0000 7fa115ffb640 10 monclient(hunting): my global_id is 4513 2024-03-25T14:26:12.817 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.812+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104004eb0 con 0x7fa1201430c0 2024-03-25T14:26:12.821 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.817+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261096214 0 0) 0x7fa120139990 con 0x7fa1200a98f0 2024-03-25T14:26:12.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.817+0000 7fa115ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:26:12.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.817+0000 7fa115ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:12.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.817+0000 7fa115ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:12.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.817+0000 7fa115ffb640 10 monclient(hunting): my global_id is 4509 2024-03-25T14:26:12.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.817+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1201d7880 con 0x7fa1200a98f0 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.823+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 441382495 0 0) 0x7fa104004eb0 con 0x7fa1201430c0 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.823+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa120139990 con 0x7fa1201430c0 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.823+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa118001c30 con 0x7fa1201430c0 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.823+0000 7fa115ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.823+0000 7fa115ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.823+0000 7fa115ffb640 10 monclient(hunting): dump: 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:12.828 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.823+0000 7fa115ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.825+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108480952 0 0) 0x7fa1201d84b0 con 0x7fa12013f710 2024-03-25T14:26:12.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.825+0000 7fa115ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:26:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.825+0000 7fa115ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.825+0000 7fa115ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.825+0000 7fa115ffb640 10 monclient(hunting): my global_id is 4487 2024-03-25T14:26:12.830 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.825+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104004eb0 con 0x7fa12013f710 2024-03-25T14:26:12.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3723919482 0 0) 0x7fa120139990 con 0x7fa1201430c0 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 >> v1:172.21.15.63:6790/0 conn(0x7fa12013f710 legacy=0x7fa1201d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 >> v1:172.21.15.63:6789/0 conn(0x7fa1200a98f0 legacy=0x7fa1201d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 1 monclient: found mon.b 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1201d5000 con 0x7fa1201430c0 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 10 monclient: _finish_auth 0 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa115ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 5 monclient: authenticate success, global_id 4513 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 10 monclient: _renew_subs 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:12.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1201d52b0 con 0x7fa1201430c0 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 10 monclient: _renew_subs 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1201d43c0 con 0x7fa1201430c0 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.834+0000 7fa12797b640 1 librados: init done 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.835+0000 7fa12797b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.835+0000 7fa12797b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:12.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.835+0000 7fa12797b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:12.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:12.835+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1201d76e0 con 0x7fa1201430c0 2024-03-25T14:26:13.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.180+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa118003990 con 0x7fa1201430c0 2024-03-25T14:26:13.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.180+0000 7fa115ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:13.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.180+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa118003c00 con 0x7fa1201430c0 2024-03-25T14:26:13.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.180+0000 7fa115ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:13.185 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.180+0000 7fa115ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.180+0000 7fa115ffb640 10 monclient: dump: 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.180+0000 7fa1177fe640 4 set_mon_vals no callback set 2024-03-25T14:26:13.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.181+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fa118003e20 con 0x7fa1201430c0 2024-03-25T14:26:13.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.182+0000 7fa125717640 10 monclient: get_auth_request con 0x7fa10406de00 auth_method 0 2024-03-25T14:26:13.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.182+0000 7fa115ffb640 10 client.4513.objecter ms_handle_connect 0x7fa10406de00 2024-03-25T14:26:13.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.184+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fa118054900 con 0x7fa1201430c0 2024-03-25T14:26:13.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.184+0000 7fa115ffb640 10 client.4513.objecter ms_dispatch 0x7fa120000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:13.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.184+0000 7fa115ffb640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:13.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.184+0000 7fa115ffb640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:13.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.184+0000 7fa115ffb640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T14:26:13.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.187+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa118050410 con 0x7fa1201430c0 2024-03-25T14:26:13.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.187+0000 7fa115ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:13.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.187+0000 7fa115ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:13.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.420+0000 7fa12797b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:13.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.420+0000 7fa12797b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:13.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.420+0000 7fa12797b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:13.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.420+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1201d7a10 con 0x7fa1201430c0 2024-03-25T14:26:13.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.568+0000 7fa115ffb640 1 -- 172.21.15.63:0/1562242143 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fa118052340 con 0x7fa1201430c0 2024-03-25T14:26:13.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.568+0000 7fa115ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:13.572 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.568+0000 7fa115ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:13.573 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:13.573 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:13.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.570+0000 7fa12797b640 10 librados: watch_flush enter 2024-03-25T14:26:13.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 10 librados: watch_flush exit 2024-03-25T14:26:13.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T14:26:13.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 10 client.4513.objecter successfully canceled tick 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 >> v1:172.21.15.153:6800/1688255552 conn(0x7fa10406de00 legacy=0x7fa1040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 10 monclient: shutdown 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 >> v1:172.21.15.153:6789/0 conn(0x7fa1201430c0 legacy=0x7fa120138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 shutdown_connections 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 >> 172.21.15.63:0/1562242143 conn(0x7fa1200b7ea0 msgr2=0x7fa120074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 shutdown_connections 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 1 -- 172.21.15.63:0/1562242143 wait complete. 2024-03-25T14:26:13.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:13.571+0000 7fa12797b640 1 librados: shutdown 2024-03-25T14:26:13.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:20.394 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:20.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.502+0000 7f010af4d640 10 monclient: get_monmap_and_config 2024-03-25T14:26:20.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.502+0000 7f010af4d640 10 monclient: build_initial_monmap 2024-03-25T14:26:20.506 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.502+0000 7f010af4d640 10 monclient: monmap: 2024-03-25T14:26:20.506 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:20.506 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:20.503401+0000 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:20.503401+0000 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:20.507 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:20.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.503+0000 7f010af4d640 1 Processor -- start 2024-03-25T14:26:20.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.503+0000 7f010af4d640 1 -- start start 2024-03-25T14:26:20.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.503+0000 7f010af4d640 10 monclient: init 2024-03-25T14:26:20.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.503+0000 7f010af4d640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.503+0000 7f010af4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 10 monclient(hunting): picked mon.noname-c con 0x7f01040b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 10 monclient(hunting): picked mon.noname-a con 0x7f01040b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 10 monclient(hunting): picked mon.noname-b con 0x7f010414be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01040af940 con 0x7f01040b1730 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01040b0a70 con 0x7f01040b4d60 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01040b0d50 con 0x7f010414be90 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:20.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.504+0000 7f010af4d640 10 monclient(hunting): authenticate will time out at 1743.648630s 2024-03-25T14:26:20.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.510+0000 7f0108ce9640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f01040b4d60 0x7f010413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:56848/0 (socket says 172.21.15.63:56848) 2024-03-25T14:26:20.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.510+0000 7f0108ce9640 1 -- 172.21.15.63:0/1870729223 learned_addr learned my addr 172.21.15.63:0/1870729223 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:20.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.521+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871832251 0 0) 0x7f01040b0a70 con 0x7f01040b4d60 2024-03-25T14:26:20.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.521+0000 7f01037fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:20.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.521+0000 7f01037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:20.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.521+0000 7f01037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:20.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.521+0000 7f01037fe640 10 monclient(hunting): my global_id is 4516 2024-03-25T14:26:20.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.521+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f4003520 con 0x7f01040b4d60 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1424230652 0 0) 0x7f00f4003520 con 0x7f01040b4d60 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 >> v1:172.21.15.63:6790/0 conn(0x7f010414be90 legacy=0x7f010414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 >> v1:172.21.15.63:6789/0 conn(0x7f01040b1730 legacy=0x7f01040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 10 monclient: _finish_hunting 0 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 1 monclient: found mon.noname-c 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01040b1030 con 0x7f01040b4d60 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 10 monclient: _finish_auth 0 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 10 monclient: _check_auth_tickets 2024-03-25T14:26:20.536 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f00f8001690 con 0x7f01040b4d60 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.531+0000 7f01037fe640 10 monclient: dump: 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:20.537 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:20.538 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:20.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.532+0000 7f010af4d640 5 monclient: authenticate success, global_id 4516 2024-03-25T14:26:20.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.532+0000 7f010af4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:20.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00f80017f0 con 0x7f01040b4d60 2024-03-25T14:26:20.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f01037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:20.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f01037fe640 1 -- 172.21.15.63:0/1870729223 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f00f8001c00 con 0x7f01040b4d60 2024-03-25T14:26:20.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f010af4d640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f010af4d640 4 set_mon_vals no callback set 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f01037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f01037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.539+0000 7f01037fe640 10 monclient: dump: 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.544 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.545 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:20.545 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:20.545 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:20.545 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:20.545 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:20.545 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:20.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.540+0000 7f010af4d640 10 monclient: shutdown 2024-03-25T14:26:20.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.540+0000 7f010af4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:20.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.540+0000 7f010af4d640 1 -- 172.21.15.63:0/1870729223 >> v1:172.21.15.153:6789/0 conn(0x7f01040b4d60 legacy=0x7f010413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.540+0000 7f010af4d640 1 -- 172.21.15.63:0/1870729223 shutdown_connections 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.540+0000 7f010af4d640 1 -- 172.21.15.63:0/1870729223 >> 172.21.15.63:0/1870729223 conn(0x7f01040617a0 msgr2=0x7f0104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.540+0000 7f010af4d640 1 -- 172.21.15.63:0/1870729223 shutdown_connections 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.540+0000 7f010af4d640 1 -- 172.21.15.63:0/1870729223 wait complete. 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.541+0000 7f010af4d640 10 monclient: build_initial_monmap 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.541+0000 7f010af4d640 10 monclient: monmap: 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:20.548 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.542+0000 7f010af4d640 1 librados: starting msgr at 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.542+0000 7f010af4d640 1 librados: starting objecter 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.542+0000 7f010af4d640 1 Processor -- start 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.542+0000 7f010af4d640 1 -- start start 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.542+0000 7f010af4d640 1 librados: setting wanted keys 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.542+0000 7f010af4d640 1 librados: calling monclient init 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.542+0000 7f010af4d640 10 monclient: init 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 10 monclient(hunting): picked mon.noname-a con 0x7f01040b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:20.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 10 monclient(hunting): picked mon.noname-b con 0x7f01040b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:20.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 10 monclient(hunting): picked mon.noname-c con 0x7f010414be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:20.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01041395b0 con 0x7f01040b1730 2024-03-25T14:26:20.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010413ae80 con 0x7f01040b4d60 2024-03-25T14:26:20.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0104139bc0 con 0x7f010414be90 2024-03-25T14:26:20.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:20.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.543+0000 7f010af4d640 10 monclient(hunting): authenticate will time out at 1743.687761s 2024-03-25T14:26:20.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.549+0000 7f0103fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f01040b4d60 0x7f010413b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:56870/0 (socket says 172.21.15.63:56870) 2024-03-25T14:26:20.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.550+0000 7f0103fff640 1 -- 172.21.15.63:0/1660796262 learned_addr learned my addr 172.21.15.63:0/1660796262 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:20.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.551+0000 7f01017fa640 10 client.?.objecter ms_handle_connect 0x7f01040b4d60 2024-03-25T14:26:20.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.551+0000 7f01017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:20.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.556+0000 7f01017fa640 10 client.?.objecter ms_handle_connect 0x7f010414be90 2024-03-25T14:26:20.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.556+0000 7f01017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:20.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.556+0000 7f01017fa640 10 client.?.objecter ms_handle_connect 0x7f01040b1730 2024-03-25T14:26:20.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.556+0000 7f01017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:20.564 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.560+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000138381 0 0) 0x7f010413ae80 con 0x7f01040b4d60 2024-03-25T14:26:20.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.560+0000 7f01017fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:20.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.560+0000 7f01017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:20.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.560+0000 7f01017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:20.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.560+0000 7f01017fa640 10 monclient(hunting): my global_id is 4519 2024-03-25T14:26:20.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.560+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e8004eb0 con 0x7f01040b4d60 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246513945 0 0) 0x7f01041395b0 con 0x7f01040b1730 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): my global_id is 4515 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010413ae80 con 0x7f01040b1730 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964481879 0 0) 0x7f0104139bc0 con 0x7f010414be90 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:20.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:20.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 10 monclient(hunting): my global_id is 4493 2024-03-25T14:26:20.568 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.562+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01041395b0 con 0x7f010414be90 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.572+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2034942554 0 0) 0x7f01041395b0 con 0x7f010414be90 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.572+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0104139bc0 con 0x7f010414be90 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.572+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f00fc001a90 con 0x7f010414be90 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.572+0000 7f01017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.572+0000 7f01017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.572+0000 7f01017fa640 10 monclient(hunting): dump: 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.576 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.577 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:20.577 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:20.577 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:20.577 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:20.577 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:20.577 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:20.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.572+0000 7f01017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:20.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.577+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1095694692 0 0) 0x7f010413ae80 con 0x7f01040b1730 2024-03-25T14:26:20.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.577+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01041395b0 con 0x7f01040b1730 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.577+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f00f8001b00 con 0x7f01040b1730 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.577+0000 7f01017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.577+0000 7f01017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.577+0000 7f01017fa640 10 monclient(hunting): dump: 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:20.582 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:20.583 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:20.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2587297549 0 0) 0x7f0104139bc0 con 0x7f010414be90 2024-03-25T14:26:20.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 >> v1:172.21.15.153:6789/0 conn(0x7f01040b4d60 legacy=0x7f010413b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:20.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 >> v1:172.21.15.63:6789/0 conn(0x7f01040b1730 legacy=0x7f010413b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 10 monclient: _finish_hunting 0 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 1 monclient: found mon.c 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0104139d50 con 0x7f010414be90 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 10 monclient: _finish_auth 0 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 10 monclient: _check_auth_tickets 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f01017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 5 monclient: authenticate success, global_id 4493 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 10 monclient: _renew_subs 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0104139000 con 0x7f010414be90 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:20.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 10 monclient: _renew_subs 2024-03-25T14:26:20.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:20.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0104143270 con 0x7f010414be90 2024-03-25T14:26:20.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.582+0000 7f010af4d640 1 librados: init done 2024-03-25T14:26:20.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.583+0000 7f010af4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:20.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.583+0000 7f010af4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:20.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.583+0000 7f010af4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:20.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.583+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0104139350 con 0x7f010414be90 2024-03-25T14:26:20.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00fc001fa0 con 0x7f010414be90 2024-03-25T14:26:20.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f01017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:20.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f00fc003b10 con 0x7f010414be90 2024-03-25T14:26:20.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f01017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:20.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f01017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:26:20.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f01017fa640 10 monclient: dump: 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:20.891 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f0102ffd640 4 set_mon_vals no callback set 2024-03-25T14:26:20.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.887+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f00fc003d30 con 0x7f010414be90 2024-03-25T14:26:20.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.888+0000 7f0108ce9640 10 monclient: get_auth_request con 0x7f00e806dfb0 auth_method 0 2024-03-25T14:26:20.892 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.889+0000 7f01017fa640 10 client.4493.objecter ms_handle_connect 0x7f00e806dfb0 2024-03-25T14:26:20.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f00fc053700 con 0x7f010414be90 2024-03-25T14:26:20.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 10 client.4493.objecter ms_dispatch 0x7f0104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:20.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:20.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:20.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-25T14:26:20.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00fc04f210 con 0x7f010414be90 2024-03-25T14:26:20.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:20.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:20.894+0000 7f01017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:21.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.149+0000 7f010af4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:21.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.149+0000 7f010af4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:21.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.149+0000 7f010af4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:21.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.149+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0104139760 con 0x7f010414be90 2024-03-25T14:26:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.173+0000 7f01017fa640 1 -- 172.21.15.63:0/1660796262 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f0104139760 con 0x7f010414be90 2024-03-25T14:26:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.173+0000 7f01017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:21.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.173+0000 7f01017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:21.177 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:21.177 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:21.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.175+0000 7f010af4d640 10 librados: watch_flush enter 2024-03-25T14:26:21.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.175+0000 7f010af4d640 10 librados: watch_flush exit 2024-03-25T14:26:21.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.175+0000 7f010af4d640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-25T14:26:21.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.175+0000 7f010af4d640 10 client.4493.objecter successfully canceled tick 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 >> v1:172.21.15.153:6800/1688255552 conn(0x7f00e806dfb0 legacy=0x7f00e8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 10 monclient: shutdown 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 >> v1:172.21.15.63:6790/0 conn(0x7f010414be90 legacy=0x7f01041e26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 shutdown_connections 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 >> 172.21.15.63:0/1660796262 conn(0x7f01040617a0 msgr2=0x7f01040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 shutdown_connections 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 1 -- 172.21.15.63:0/1660796262 wait complete. 2024-03-25T14:26:21.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:21.176+0000 7f010af4d640 1 librados: shutdown 2024-03-25T14:26:21.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:27.220 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:05.732 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.229 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:05.741 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.253 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:05.765 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.282 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:05.742 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.288 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:05.748 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.291 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:05.804 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.299 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:05.811 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.329 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:05.789 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.340 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:05.853 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:27.998 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:28.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.107+0000 7f261c0b5640 10 monclient: get_monmap_and_config 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.107+0000 7f261c0b5640 10 monclient: build_initial_monmap 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.107+0000 7f261c0b5640 10 monclient: monmap: 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:28.107904+0000 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:28.107904+0000 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:28.111 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.108+0000 7f261c0b5640 1 Processor -- start 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.108+0000 7f261c0b5640 1 -- start start 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.108+0000 7f261c0b5640 10 monclient: init 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 10 monclient(hunting): picked mon.noname-c con 0x7f26140a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 10 monclient(hunting): picked mon.noname-a con 0x7f261413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:28.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 10 monclient(hunting): picked mon.noname-b con 0x7f26141430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:28.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2614142520 con 0x7f261413f710 2024-03-25T14:26:28.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26140b28a0 con 0x7f26140a98f0 2024-03-25T14:26:28.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26140b2a30 con 0x7f26141430c0 2024-03-25T14:26:28.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:28.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.109+0000 7f261c0b5640 10 monclient(hunting): authenticate will time out at 1751.253111s 2024-03-25T14:26:28.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.118+0000 7f2619e51640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f26140a98f0 0x7f26140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:33060/0 (socket says 172.21.15.63:33060) 2024-03-25T14:26:28.121 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.118+0000 7f2619e51640 1 -- 172.21.15.63:0/2878144087 learned_addr learned my addr 172.21.15.63:0/2878144087 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:28.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.134+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143715593 0 0) 0x7f26140b28a0 con 0x7f26140a98f0 2024-03-25T14:26:28.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.134+0000 7f2618e4f640 10 monclient(hunting): _negotiate 2024-03-25T14:26:28.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.134+0000 7f2618e4f640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:28.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.134+0000 7f2618e4f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:28.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.134+0000 7f2618e4f640 10 monclient(hunting): my global_id is 4522 2024-03-25T14:26:28.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.134+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2600003520 con 0x7f26140a98f0 2024-03-25T14:26:28.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3662400646 0 0) 0x7f2600003520 con 0x7f26140a98f0 2024-03-25T14:26:28.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 >> v1:172.21.15.63:6790/0 conn(0x7f26141430c0 legacy=0x7f261414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:28.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 >> v1:172.21.15.63:6789/0 conn(0x7f261413f710 legacy=0x7f26141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:28.144 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 10 monclient: _finish_hunting 0 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 1 monclient: found mon.noname-c 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26140b44c0 con 0x7f26140a98f0 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 10 monclient: _finish_auth 0 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 10 monclient: _check_auth_tickets 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2610001690 con 0x7f26140a98f0 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f2618e4f640 10 monclient: dump: 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:28.145 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f261c0b5640 5 monclient: authenticate success, global_id 4522 2024-03-25T14:26:28.146 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.140+0000 7f261c0b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:28.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26100017f0 con 0x7f26140a98f0 2024-03-25T14:26:28.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f2618e4f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:28.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f2618e4f640 1 -- 172.21.15.63:0/2878144087 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f2610001c40 con 0x7f26140a98f0 2024-03-25T14:26:28.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f2618e4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f2618e4f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f2618e4f640 10 monclient: dump: 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:28.153 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:28.154 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:28.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f261c0b5640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:28.154 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.149+0000 7f261c0b5640 4 set_mon_vals no callback set 2024-03-25T14:26:28.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.150+0000 7f261c0b5640 10 monclient: shutdown 2024-03-25T14:26:28.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.150+0000 7f261c0b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:28.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.150+0000 7f261c0b5640 1 -- 172.21.15.63:0/2878144087 >> v1:172.21.15.153:6789/0 conn(0x7f26140a98f0 legacy=0x7f26140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:28.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.150+0000 7f261c0b5640 1 -- 172.21.15.63:0/2878144087 shutdown_connections 2024-03-25T14:26:28.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.150+0000 7f261c0b5640 1 -- 172.21.15.63:0/2878144087 >> 172.21.15.63:0/2878144087 conn(0x7f26140b7ea0 msgr2=0x7f26140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:28.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.150+0000 7f261c0b5640 1 -- 172.21.15.63:0/2878144087 shutdown_connections 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.150+0000 7f261c0b5640 1 -- 172.21.15.63:0/2878144087 wait complete. 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 10 monclient: build_initial_monmap 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 10 monclient: monmap: 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:28.157 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 1 librados: starting msgr at 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 1 librados: starting objecter 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 1 Processor -- start 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 1 -- start start 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 1 librados: setting wanted keys 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 1 librados: calling monclient init 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.151+0000 7f261c0b5640 10 monclient: init 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:26:28.158 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 10 monclient(hunting): picked mon.noname-c con 0x7f26140a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:28.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 10 monclient(hunting): picked mon.noname-a con 0x7f261413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:28.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 10 monclient(hunting): picked mon.noname-b con 0x7f26141430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:28.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261406e630 con 0x7f261413f710 2024-03-25T14:26:28.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26141dfa10 con 0x7f26141430c0 2024-03-25T14:26:28.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26141339d0 con 0x7f26140a98f0 2024-03-25T14:26:28.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:28.159 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.152+0000 7f261c0b5640 10 monclient(hunting): authenticate will time out at 1751.296206s 2024-03-25T14:26:28.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.159+0000 7f2619e51640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f26140a98f0 0x7f26141df300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:47226/0 (socket says 172.21.15.63:47226) 2024-03-25T14:26:28.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.159+0000 7f2619650640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f261413f710 0x7f2614132bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:33774/0 (socket says 172.21.15.63:33774) 2024-03-25T14:26:28.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.159+0000 7f2619e51640 1 -- 172.21.15.63:0/1819530497 learned_addr learned my addr 172.21.15.63:0/1819530497 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:28.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.161+0000 7f260a7fc640 10 client.?.objecter ms_handle_connect 0x7f26141430c0 2024-03-25T14:26:28.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.161+0000 7f260a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:28.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.161+0000 7f260a7fc640 10 client.?.objecter ms_handle_connect 0x7f26140a98f0 2024-03-25T14:26:28.164 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.161+0000 7f260a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:28.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.162+0000 7f260a7fc640 10 client.?.objecter ms_handle_connect 0x7f261413f710 2024-03-25T14:26:28.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.162+0000 7f260a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:28.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.169+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551866085 0 0) 0x7f26141dfa10 con 0x7f26141430c0 2024-03-25T14:26:28.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.169+0000 7f260a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:26:28.172 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.169+0000 7f260a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:28.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.169+0000 7f260a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:28.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.169+0000 7f260a7fc640 10 monclient(hunting): my global_id is 4525 2024-03-25T14:26:28.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.169+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f8004eb0 con 0x7f26141430c0 2024-03-25T14:26:28.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.172+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072834217 0 0) 0x7f26141339d0 con 0x7f26140a98f0 2024-03-25T14:26:28.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.173+0000 7f260a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:26:28.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.173+0000 7f260a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:28.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.173+0000 7f260a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:28.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.173+0000 7f260a7fc640 10 monclient(hunting): my global_id is 4499 2024-03-25T14:26:28.176 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.173+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26141dfa10 con 0x7f26140a98f0 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3002028198 0 0) 0x7f25f8004eb0 con 0x7f26141430c0 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26141339d0 con 0x7f26141430c0 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f260c001b90 con 0x7f26141430c0 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): dump: 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.180 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353175836 0 0) 0x7f261406e630 con 0x7f261413f710 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T14:26:28.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.176+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f8004eb0 con 0x7f261413f710 2024-03-25T14:26:28.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2108459360 0 0) 0x7f26141339d0 con 0x7f26141430c0 2024-03-25T14:26:28.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 >> v1:172.21.15.63:6790/0 conn(0x7f26140a98f0 legacy=0x7f26141df300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:28.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 >> v1:172.21.15.63:6789/0 conn(0x7f261413f710 legacy=0x7f2614132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:28.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 1 monclient: found mon.b 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261406f370 con 0x7f26141430c0 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f260a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 5 monclient: authenticate success, global_id 4525 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 10 monclient: _renew_subs 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261406f650 con 0x7f26141430c0 2024-03-25T14:26:28.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 10 monclient: _renew_subs 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26140b7840 con 0x7f26141430c0 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.192+0000 7f261c0b5640 1 librados: init done 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.193+0000 7f261c0b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.193+0000 7f261c0b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.193+0000 7f261c0b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:28.198 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.193+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26140b74b0 con 0x7f26141430c0 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.469+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f260c002080 con 0x7f26141430c0 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.469+0000 7f260a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.469+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f260c003b30 con 0x7f26141430c0 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.469+0000 7f260a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.469+0000 7f260a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.469+0000 7f260a7fc640 10 monclient: dump: 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.473 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:28.474 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:28.474 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:28.474 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:28.474 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:28.474 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:28.474 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:28.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.469+0000 7f260bfff640 4 set_mon_vals no callback set 2024-03-25T14:26:28.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.470+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f260c003d50 con 0x7f26141430c0 2024-03-25T14:26:28.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.471+0000 7f2619e51640 10 monclient: get_auth_request con 0x7f25f806ddb0 auth_method 0 2024-03-25T14:26:28.475 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.471+0000 7f260a7fc640 10 client.4525.objecter ms_handle_connect 0x7f25f806ddb0 2024-03-25T14:26:28.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.474+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f260c053910 con 0x7f26141430c0 2024-03-25T14:26:28.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.474+0000 7f260a7fc640 10 client.4525.objecter ms_dispatch 0x7f2614000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:28.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.474+0000 7f260a7fc640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:28.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.474+0000 7f260a7fc640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:28.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.474+0000 7f260a7fc640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T14:26:28.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.475+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f260c04f250 con 0x7f26141430c0 2024-03-25T14:26:28.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.475+0000 7f260a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:28.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.475+0000 7f260a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:28.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.735+0000 7f261c0b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:28.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.735+0000 7f261c0b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:28.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.735+0000 7f261c0b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:28.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.735+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f261413e320 con 0x7f26141430c0 2024-03-25T14:26:28.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.758+0000 7f260a7fc640 1 -- 172.21.15.63:0/1819530497 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f260c051350 con 0x7f26141430c0 2024-03-25T14:26:28.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.758+0000 7f260a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:28.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.758+0000 7f260a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:28.762 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:28.762 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 10 librados: watch_flush enter 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 10 librados: watch_flush exit 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 10 client.4525.objecter successfully canceled tick 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 >> v1:172.21.15.153:6800/1688255552 conn(0x7f25f806ddb0 legacy=0x7f25f8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 10 monclient: shutdown 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 >> v1:172.21.15.153:6789/0 conn(0x7f26141430c0 legacy=0x7f26141332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 shutdown_connections 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 >> 172.21.15.63:0/1819530497 conn(0x7f26140b7ea0 msgr2=0x7f2614074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 shutdown_connections 2024-03-25T14:26:28.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 1 -- 172.21.15.63:0/1819530497 wait complete. 2024-03-25T14:26:28.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:28.761+0000 7f261c0b5640 1 librados: shutdown 2024-03-25T14:26:28.776 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:35.583 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:35.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.693+0000 7faffa606640 10 monclient: get_monmap_and_config 2024-03-25T14:26:35.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.693+0000 7faffa606640 10 monclient: build_initial_monmap 2024-03-25T14:26:35.698 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.693+0000 7faffa606640 10 monclient: monmap: 2024-03-25T14:26:35.698 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:35.698 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:35.698 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:35.695021+0000 2024-03-25T14:26:35.699 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:35.695021+0000 2024-03-25T14:26:35.699 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:35.699 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:35.699 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:35.699 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:35.699 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:35.699 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.694+0000 7faffa606640 1 Processor -- start 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.695+0000 7faffa606640 1 -- start start 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.695+0000 7faffa606640 10 monclient: init 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.695+0000 7faffa606640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.695+0000 7faffa606640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 10 monclient(hunting): picked mon.noname-b con 0x7faff406e200 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 10 monclient(hunting): picked mon.noname-c con 0x7faff4072730 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 10 monclient(hunting): picked mon.noname-a con 0x7faff406e5e0 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:35.700 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff40ae470 con 0x7faff406e5e0 2024-03-25T14:26:35.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff40b0320 con 0x7faff4072730 2024-03-25T14:26:35.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff406efa0 con 0x7faff406e200 2024-03-25T14:26:35.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:35.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.696+0000 7faffa606640 10 monclient(hunting): authenticate will time out at 1758.840610s 2024-03-25T14:26:35.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.704+0000 7faff37fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7faff4072730 0x7faff4146bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:52596/0 (socket says 172.21.15.63:52596) 2024-03-25T14:26:35.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.704+0000 7faff37fe640 1 -- 172.21.15.63:0/3045434023 learned_addr learned my addr 172.21.15.63:0/3045434023 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:35.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.717+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984307696 0 0) 0x7faff40b0320 con 0x7faff4072730 2024-03-25T14:26:35.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.717+0000 7faff2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:26:35.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.717+0000 7faff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:35.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.717+0000 7faff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:35.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.717+0000 7faff2ffd640 10 monclient(hunting): my global_id is 4528 2024-03-25T14:26:35.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.717+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafdc003520 con 0x7faff4072730 2024-03-25T14:26:35.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.724+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111945459 0 0) 0x7faff40ae470 con 0x7faff406e5e0 2024-03-25T14:26:35.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.724+0000 7faff2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:26:35.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.724+0000 7faff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:35.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.724+0000 7faff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:35.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.724+0000 7faff2ffd640 10 monclient(hunting): my global_id is 4524 2024-03-25T14:26:35.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.724+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff40b0320 con 0x7faff406e5e0 2024-03-25T14:26:35.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.727+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857348641 0 0) 0x7faff406efa0 con 0x7faff406e200 2024-03-25T14:26:35.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.727+0000 7faff2ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:26:35.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.727+0000 7faff2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:35.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.727+0000 7faff2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:35.731 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.727+0000 7faff2ffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T14:26:35.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.727+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff40ae470 con 0x7faff406e200 2024-03-25T14:26:35.746 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2011774324 0 0) 0x7fafdc003520 con 0x7faff4072730 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 >> v1:172.21.15.63:6790/0 conn(0x7faff406e200 legacy=0x7faff4072020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 >> v1:172.21.15.63:6789/0 conn(0x7faff406e5e0 legacy=0x7faff4152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 1 monclient: found mon.noname-c 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff406f3f0 con 0x7faff4072730 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 10 monclient: _finish_auth 0 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fafe8001690 con 0x7faff4072730 2024-03-25T14:26:35.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faffa606640 5 monclient: authenticate success, global_id 4528 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faffa606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faff2ffd640 10 monclient: dump: 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:35.748 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:35.749 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:35.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.742+0000 7faffa606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:35.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe80017f0 con 0x7faff4072730 2024-03-25T14:26:35.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faff2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:35.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faff2ffd640 1 -- 172.21.15.63:0/3045434023 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fafe8001c00 con 0x7faff4072730 2024-03-25T14:26:35.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faffa606640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:35.761 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faffa606640 4 set_mon_vals no callback set 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faff2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faff2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.757+0000 7faff2ffd640 10 monclient: dump: 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:35.762 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.758+0000 7faffa606640 10 monclient: shutdown 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.758+0000 7faffa606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.758+0000 7faffa606640 1 -- 172.21.15.63:0/3045434023 >> v1:172.21.15.153:6789/0 conn(0x7faff4072730 legacy=0x7faff4146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.758+0000 7faffa606640 1 -- 172.21.15.63:0/3045434023 shutdown_connections 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.758+0000 7faffa606640 1 -- 172.21.15.63:0/3045434023 >> 172.21.15.63:0/3045434023 conn(0x7faff406b630 msgr2=0x7faff40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.758+0000 7faffa606640 1 -- 172.21.15.63:0/3045434023 shutdown_connections 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.758+0000 7faffa606640 1 -- 172.21.15.63:0/3045434023 wait complete. 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 10 monclient: build_initial_monmap 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 10 monclient: monmap: 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:35.764 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 1 librados: starting msgr at 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 1 librados: starting objecter 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 1 Processor -- start 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 1 -- start start 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 1 librados: setting wanted keys 2024-03-25T14:26:35.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 1 librados: calling monclient init 2024-03-25T14:26:35.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.759+0000 7faffa606640 10 monclient: init 2024-03-25T14:26:35.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 10 monclient(hunting): picked mon.noname-b con 0x7faff406e200 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 10 monclient(hunting): picked mon.noname-c con 0x7faff406e5e0 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 10 monclient(hunting): picked mon.noname-a con 0x7faff4072730 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff41402a0 con 0x7faff4072730 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff40ad3a0 con 0x7faff406e200 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff40aab20 con 0x7faff406e5e0 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:35.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.760+0000 7faffa606640 10 monclient(hunting): authenticate will time out at 1758.905306s 2024-03-25T14:26:35.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.765+0000 7faff37fe640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7faff406e5e0 0x7faff41db840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:43570/0 (socket says 172.21.15.63:43570) 2024-03-25T14:26:35.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.765+0000 7faff37fe640 1 -- 172.21.15.63:0/226274699 learned_addr learned my addr 172.21.15.63:0/226274699 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:35.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.768+0000 7faff0ff9640 10 client.?.objecter ms_handle_connect 0x7faff406e5e0 2024-03-25T14:26:35.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.768+0000 7faff0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:35.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.775+0000 7faff0ff9640 10 client.?.objecter ms_handle_connect 0x7faff4072730 2024-03-25T14:26:35.779 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.775+0000 7faff0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:35.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.777+0000 7faff0ff9640 10 client.?.objecter ms_handle_connect 0x7faff406e200 2024-03-25T14:26:35.781 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.777+0000 7faff0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:35.782 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.778+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129448537 0 0) 0x7faff40aab20 con 0x7faff406e5e0 2024-03-25T14:26:35.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.778+0000 7faff0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:26:35.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.778+0000 7faff0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:35.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.778+0000 7faff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:35.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.778+0000 7faff0ff9640 10 monclient(hunting): my global_id is 4505 2024-03-25T14:26:35.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.778+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe0004eb0 con 0x7faff406e5e0 2024-03-25T14:26:35.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.781+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384441619 0 0) 0x7faff41402a0 con 0x7faff4072730 2024-03-25T14:26:35.785 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.781+0000 7faff0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:26:35.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.781+0000 7faff0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:35.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.781+0000 7faff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:35.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.781+0000 7faff0ff9640 10 monclient(hunting): my global_id is 4527 2024-03-25T14:26:35.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.781+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff40aab20 con 0x7faff4072730 2024-03-25T14:26:35.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.791+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107206912 0 0) 0x7faff40aab20 con 0x7faff4072730 2024-03-25T14:26:35.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.791+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faff41402a0 con 0x7faff4072730 2024-03-25T14:26:35.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.791+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fafec001cb0 con 0x7faff4072730 2024-03-25T14:26:35.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.791+0000 7faff0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.791+0000 7faff0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.791+0000 7faff0ff9640 10 monclient(hunting): dump: 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:35.796 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.791+0000 7faff0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:35.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.795+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4096408136 0 0) 0x7fafe0004eb0 con 0x7faff406e5e0 2024-03-25T14:26:35.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.795+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faff40aab20 con 0x7faff406e5e0 2024-03-25T14:26:35.799 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.795+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fafe8001bc0 con 0x7faff406e5e0 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.795+0000 7faff0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.795+0000 7faff0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.795+0000 7faff0ff9640 10 monclient(hunting): dump: 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:35.800 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:35.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.798+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171020487 0 0) 0x7faff40ad3a0 con 0x7faff406e200 2024-03-25T14:26:35.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.798+0000 7faff0ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:26:35.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.798+0000 7faff0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:35.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.798+0000 7faff0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:35.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.798+0000 7faff0ff9640 10 monclient(hunting): my global_id is 4531 2024-03-25T14:26:35.803 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.798+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe0004eb0 con 0x7faff406e200 2024-03-25T14:26:35.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.801+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1229323846 0 0) 0x7faff40aab20 con 0x7faff406e5e0 2024-03-25T14:26:35.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 >> v1:172.21.15.153:6789/0 conn(0x7faff406e200 legacy=0x7faff40ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:35.806 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 >> v1:172.21.15.63:6789/0 conn(0x7faff4072730 legacy=0x7faff4143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 1 monclient: found mon.c 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff41df260 con 0x7faff406e5e0 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 10 monclient: _finish_auth 0 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faff0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 5 monclient: authenticate success, global_id 4505 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 10 monclient: _renew_subs 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff4143e90 con 0x7faff406e5e0 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:35.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 10 monclient: _renew_subs 2024-03-25T14:26:35.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:35.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faff41ded40 con 0x7faff406e5e0 2024-03-25T14:26:35.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.802+0000 7faffa606640 1 librados: init done 2024-03-25T14:26:35.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.803+0000 7faffa606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:35.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.803+0000 7faffa606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:35.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.803+0000 7faffa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:35.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:35.803+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff40627b0 con 0x7faff406e5e0 2024-03-25T14:26:36.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.124+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe80011f0 con 0x7faff406e5e0 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.124+0000 7faff0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.124+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fafe8004c30 con 0x7faff406e5e0 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.124+0000 7faff0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.124+0000 7faff0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.124+0000 7faff0ff9640 10 monclient: dump: 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:36.129 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:36.130 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:36.130 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:36.130 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:36.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.124+0000 7faff1ffb640 4 set_mon_vals no callback set 2024-03-25T14:26:36.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.125+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fafe801ac40 con 0x7faff406e5e0 2024-03-25T14:26:36.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.126+0000 7faff3fff640 10 monclient: get_auth_request con 0x7fafe006deb0 auth_method 0 2024-03-25T14:26:36.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.126+0000 7faff0ff9640 10 client.4505.objecter ms_handle_connect 0x7fafe006deb0 2024-03-25T14:26:36.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fafe8054930 con 0x7faff406e5e0 2024-03-25T14:26:36.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 10 client.4505.objecter ms_dispatch 0x7faff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:36.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:36.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:36.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-25T14:26:36.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafe8050270 con 0x7faff406e5e0 2024-03-25T14:26:36.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:36.137 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.132+0000 7faff0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.401+0000 7faffa606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.401+0000 7faffa606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.401+0000 7faffa606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:36.405 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.401+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faff41df3f0 con 0x7faff406e5e0 2024-03-25T14:26:36.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.449+0000 7faff0ff9640 1 -- 172.21.15.63:0/226274699 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fafe8052370 con 0x7faff406e5e0 2024-03-25T14:26:36.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.449+0000 7faff0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:36.453 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.449+0000 7faff0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:36.454 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:36.454 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 10 librados: watch_flush enter 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 10 librados: watch_flush exit 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 10 client.4505.objecter successfully canceled tick 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 >> v1:172.21.15.153:6800/1688255552 conn(0x7fafe006deb0 legacy=0x7fafe0070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 10 monclient: shutdown 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 >> v1:172.21.15.63:6790/0 conn(0x7faff406e5e0 legacy=0x7faff41db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 shutdown_connections 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 >> 172.21.15.63:0/226274699 conn(0x7faff406b630 msgr2=0x7faff4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:36.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 shutdown_connections 2024-03-25T14:26:36.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.451+0000 7faffa606640 1 -- 172.21.15.63:0/226274699 wait complete. 2024-03-25T14:26:36.457 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:36.452+0000 7faffa606640 1 librados: shutdown 2024-03-25T14:26:36.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:43.274 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:43.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.369+0000 7f665dd82640 10 monclient: get_monmap_and_config 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.369+0000 7f665dd82640 10 monclient: build_initial_monmap 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.369+0000 7f665dd82640 10 monclient: monmap: 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:43.370704+0000 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:43.370704+0000 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:43.374 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.370+0000 7f665dd82640 1 Processor -- start 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.370+0000 7f665dd82640 1 -- start start 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.370+0000 7f665dd82640 10 monclient: init 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.370+0000 7f665dd82640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.370+0000 7f665dd82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.370+0000 7f665dd82640 10 monclient(hunting): picked mon.noname-c con 0x7f66580b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.370+0000 7f665dd82640 10 monclient(hunting): picked mon.noname-a con 0x7f66580b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f665dd82640 10 monclient(hunting): picked mon.noname-b con 0x7f665814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f665dd82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66580af940 con 0x7f66580b1730 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f665dd82640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66580b0a70 con 0x7f66580b4d60 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f665dd82640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66580b0d50 con 0x7f665814be90 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f665dd82640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f665dd82640 10 monclient(hunting): authenticate will time out at 1766.515568s 2024-03-25T14:26:43.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f66577fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f66580b4d60 0x7f665813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:41078/0 (socket says 172.21.15.63:41078) 2024-03-25T14:26:43.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.371+0000 7f66577fe640 1 -- 172.21.15.63:0/1572073227 learned_addr learned my addr 172.21.15.63:0/1572073227 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:43.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.403+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206647719 0 0) 0x7f66580af940 con 0x7f66580b1730 2024-03-25T14:26:43.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.403+0000 7f66567fc640 10 monclient(hunting): _negotiate 2024-03-25T14:26:43.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.403+0000 7f66567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:43.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.403+0000 7f66567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:43.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.403+0000 7f66567fc640 10 monclient(hunting): my global_id is 4530 2024-03-25T14:26:43.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.403+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6640003520 con 0x7f66580b1730 2024-03-25T14:26:43.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 535301608 0 0) 0x7f6640003520 con 0x7f66580b1730 2024-03-25T14:26:43.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 >> v1:172.21.15.63:6790/0 conn(0x7f665814be90 legacy=0x7f665814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:43.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 >> v1:172.21.15.153:6789/0 conn(0x7f66580b4d60 legacy=0x7f665813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:43.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 10 monclient: _finish_hunting 0 2024-03-25T14:26:43.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 1 monclient: found mon.noname-a 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66580b1030 con 0x7f66580b1730 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 10 monclient: _finish_auth 0 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 10 monclient: _check_auth_tickets 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f664c001690 con 0x7f66580b1730 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f665dd82640 5 monclient: authenticate success, global_id 4530 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f665dd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:43.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f66567fc640 10 monclient: dump: 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:43.422 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:43.423 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:43.423 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.415+0000 7f665dd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:43.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f664c0017f0 con 0x7f66580b1730 2024-03-25T14:26:43.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f66567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:43.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f66567fc640 1 -- 172.21.15.63:0/1572073227 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f664c001c00 con 0x7f66580b1730 2024-03-25T14:26:43.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f66567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:43.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f66567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f66567fc640 10 monclient: dump: 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:43.432 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:43.433 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f665dd82640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:43.433 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f665dd82640 4 set_mon_vals no callback set 2024-03-25T14:26:43.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f665dd82640 10 monclient: shutdown 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f665dd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.427+0000 7f665dd82640 1 -- 172.21.15.63:0/1572073227 >> v1:172.21.15.63:6789/0 conn(0x7f66580b1730 legacy=0x7f66580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.428+0000 7f665dd82640 1 -- 172.21.15.63:0/1572073227 shutdown_connections 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.428+0000 7f665dd82640 1 -- 172.21.15.63:0/1572073227 >> 172.21.15.63:0/1572073227 conn(0x7f66580617a0 msgr2=0x7f6658061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.428+0000 7f665dd82640 1 -- 172.21.15.63:0/1572073227 shutdown_connections 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.428+0000 7f665dd82640 1 -- 172.21.15.63:0/1572073227 wait complete. 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.428+0000 7f665dd82640 10 monclient: build_initial_monmap 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.428+0000 7f665dd82640 10 monclient: monmap: 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:43.436 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.429+0000 7f665dd82640 1 librados: starting msgr at 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.429+0000 7f665dd82640 1 librados: starting objecter 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.429+0000 7f665dd82640 1 Processor -- start 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.429+0000 7f665dd82640 1 -- start start 2024-03-25T14:26:43.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.429+0000 7f665dd82640 1 librados: setting wanted keys 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.429+0000 7f665dd82640 1 librados: calling monclient init 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.429+0000 7f665dd82640 10 monclient: init 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 10 monclient(hunting): picked mon.noname-a con 0x7f66580b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 10 monclient(hunting): picked mon.noname-b con 0x7f66580b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 10 monclient(hunting): picked mon.noname-c con 0x7f665814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66581d78d0 con 0x7f66580b1730 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665813bfa0 con 0x7f66580b4d60 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66581d84c0 con 0x7f665814be90 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:43.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.430+0000 7f665dd82640 10 monclient(hunting): authenticate will time out at 1766.574833s 2024-03-25T14:26:43.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.442+0000 7f66577fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f66580b1730 0x7f66581405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:42436/0 (socket says 172.21.15.63:42436) 2024-03-25T14:26:43.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.442+0000 7f66577fe640 1 -- 172.21.15.63:0/1959113598 learned_addr learned my addr 172.21.15.63:0/1959113598 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:43.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.443+0000 7f663ffff640 10 client.?.objecter ms_handle_connect 0x7f66580b1730 2024-03-25T14:26:43.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.443+0000 7f663ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:43.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.450+0000 7f663ffff640 10 client.?.objecter ms_handle_connect 0x7f66580b4d60 2024-03-25T14:26:43.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.450+0000 7f663ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:43.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.456+0000 7f663ffff640 10 client.?.objecter ms_handle_connect 0x7f665814be90 2024-03-25T14:26:43.460 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.456+0000 7f663ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:43.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.465+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542374433 0 0) 0x7f66581d78d0 con 0x7f66580b1730 2024-03-25T14:26:43.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.465+0000 7f663ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:26:43.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.465+0000 7f663ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:43.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.465+0000 7f663ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:43.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.465+0000 7f663ffff640 10 monclient(hunting): my global_id is 4533 2024-03-25T14:26:43.469 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.465+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6644004eb0 con 0x7f66580b1730 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209063588 0 0) 0x7f66581d84c0 con 0x7f665814be90 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): my global_id is 4511 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66581d78d0 con 0x7f665814be90 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452492039 0 0) 0x7f665813bfa0 con 0x7f66580b4d60 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 10 monclient(hunting): my global_id is 4537 2024-03-25T14:26:43.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.476+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66581d84c0 con 0x7f66580b4d60 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.487+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2696380335 0 0) 0x7f66581d84c0 con 0x7f66580b4d60 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.488+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f665813bfa0 con 0x7f66580b4d60 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.488+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f664c001bc0 con 0x7f66580b4d60 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.488+0000 7f663ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.488+0000 7f663ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.488+0000 7f663ffff640 10 monclient(hunting): dump: 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:43.492 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.488+0000 7f663ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2513988741 0 0) 0x7f66581d78d0 con 0x7f665814be90 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66581d84c0 con 0x7f665814be90 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6650001b00 con 0x7f665814be90 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 10 monclient(hunting): dump: 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:43.499 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2341834318 0 0) 0x7f6644004eb0 con 0x7f66580b1730 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66581d78d0 con 0x7f66580b1730 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f66480018b0 con 0x7f66580b1730 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.494+0000 7f663ffff640 10 monclient(hunting): dump: 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.500 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.501 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:43.501 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:43.501 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:43.501 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:43.501 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:43.501 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4223574676 0 0) 0x7f66581d84c0 con 0x7f665814be90 2024-03-25T14:26:43.507 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 >> v1:172.21.15.153:6789/0 conn(0x7f66580b4d60 legacy=0x7f66581d7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 >> v1:172.21.15.63:6789/0 conn(0x7f66580b1730 legacy=0x7f66581405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 1 monclient: found mon.c 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66581d8650 con 0x7f665814be90 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 10 monclient: _finish_auth 0 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f663ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 5 monclient: authenticate success, global_id 4511 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 10 monclient: _renew_subs 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:43.508 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6658142280 con 0x7f665814be90 2024-03-25T14:26:43.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:43.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 10 monclient: _renew_subs 2024-03-25T14:26:43.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:43.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f665813bc30 con 0x7f665814be90 2024-03-25T14:26:43.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.503+0000 7f665dd82640 1 librados: init done 2024-03-25T14:26:43.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.504+0000 7f665dd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:43.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.504+0000 7f665dd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:43.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.504+0000 7f665dd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:43.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.504+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6658142580 con 0x7f665814be90 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.525+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6650001fd0 con 0x7f665814be90 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.525+0000 7f663ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.525+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6650003970 con 0x7f665814be90 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.525+0000 7f663ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.525+0000 7f663ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.525+0000 7f663ffff640 10 monclient: dump: 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:43.529 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:43.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.525+0000 7f6655ffb640 4 set_mon_vals no callback set 2024-03-25T14:26:43.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.526+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f6650019940 con 0x7f665814be90 2024-03-25T14:26:43.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.526+0000 7f66577fe640 10 monclient: get_auth_request con 0x7f664406e110 auth_method 0 2024-03-25T14:26:43.531 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.527+0000 7f663ffff640 10 client.4511.objecter ms_handle_connect 0x7f664406e110 2024-03-25T14:26:43.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.529+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f6650053540 con 0x7f665814be90 2024-03-25T14:26:43.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.529+0000 7f663ffff640 10 client.4511.objecter ms_dispatch 0x7f6658000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:43.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.529+0000 7f663ffff640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:43.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.529+0000 7f663ffff640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:43.534 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.529+0000 7f663ffff640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-25T14:26:43.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.827+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f665004f050 con 0x7f665814be90 2024-03-25T14:26:43.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.827+0000 7f663ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:43.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:43.827+0000 7f663ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:44.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.075+0000 7f665dd82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.075+0000 7f665dd82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.075+0000 7f665dd82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:44.080 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.075+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66580627b0 con 0x7f665814be90 2024-03-25T14:26:44.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.124+0000 7f663ffff640 1 -- 172.21.15.63:0/1959113598 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f6650056ba0 con 0x7f665814be90 2024-03-25T14:26:44.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.124+0000 7f663ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:44.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.124+0000 7f663ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:44.129 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:44.129 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 10 librados: watch_flush enter 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 10 librados: watch_flush exit 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 10 client.4511.objecter successfully canceled tick 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 >> v1:172.21.15.153:6800/1688255552 conn(0x7f664406e110 legacy=0x7f66440705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 10 monclient: shutdown 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 >> v1:172.21.15.63:6790/0 conn(0x7f665814be90 legacy=0x7f6658142950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 shutdown_connections 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 >> 172.21.15.63:0/1959113598 conn(0x7f66580617a0 msgr2=0x7f66580b6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 shutdown_connections 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 1 -- 172.21.15.63:0/1959113598 wait complete. 2024-03-25T14:26:44.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:44.127+0000 7f665dd82640 1 librados: shutdown 2024-03-25T14:26:44.144 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:47.193 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:25.705 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.198 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:25.710 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.276 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:25.789 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.310 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:25.769 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.313 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:25.773 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.331 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:25.844 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.338 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:25.798 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.339 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:25.852 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:47.385 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:25.898 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:26:50.950 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:51.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.061+0000 7f02eab40640 10 monclient: get_monmap_and_config 2024-03-25T14:26:51.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.061+0000 7f02eab40640 10 monclient: build_initial_monmap 2024-03-25T14:26:51.066 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.062+0000 7f02eab40640 10 monclient: monmap: 2024-03-25T14:26:51.066 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:51.066 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:51.066 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:51.063100+0000 2024-03-25T14:26:51.066 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:51.063100+0000 2024-03-25T14:26:51.067 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:51.067 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:51.067 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:51.067 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:51.067 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:51.067 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:51.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.062+0000 7f02eab40640 1 Processor -- start 2024-03-25T14:26:51.068 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.062+0000 7f02eab40640 1 -- start start 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.062+0000 7f02eab40640 10 monclient: init 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 10 monclient(hunting): picked mon.noname-b con 0x7f02e40a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 10 monclient(hunting): picked mon.noname-a con 0x7f02e413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 10 monclient(hunting): picked mon.noname-c con 0x7f02e41430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e4142520 con 0x7f02e413f710 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40b28a0 con 0x7f02e41430c0 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40b2a30 con 0x7f02e40a98f0 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:51.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.063+0000 7f02eab40640 10 monclient(hunting): authenticate will time out at 1774.208240s 2024-03-25T14:26:51.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.074+0000 7f02e90dd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f02e41430c0 0x7f02e414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:57580/0 (socket says 172.21.15.63:57580) 2024-03-25T14:26:51.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.074+0000 7f02e90dd640 1 -- 172.21.15.63:0/1556847370 learned_addr learned my addr 172.21.15.63:0/1556847370 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:51.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.092+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003580592 0 0) 0x7f02e40b2a30 con 0x7f02e40a98f0 2024-03-25T14:26:51.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.092+0000 7f02e37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:51.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.092+0000 7f02e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:51.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.092+0000 7f02e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:51.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.092+0000 7f02e37fe640 10 monclient(hunting): my global_id is 4514 2024-03-25T14:26:51.097 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.092+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d4003520 con 0x7f02e40a98f0 2024-03-25T14:26:51.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.097+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76793415 0 0) 0x7f02e40b28a0 con 0x7f02e41430c0 2024-03-25T14:26:51.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.098+0000 7f02e37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:51.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.098+0000 7f02e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:51.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.098+0000 7f02e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:51.102 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.098+0000 7f02e37fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T14:26:51.103 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.098+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e40b2a30 con 0x7f02e41430c0 2024-03-25T14:26:51.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2318119570 0 0) 0x7f02e40b2a30 con 0x7f02e41430c0 2024-03-25T14:26:51.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 >> v1:172.21.15.63:6790/0 conn(0x7f02e40a98f0 legacy=0x7f02e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 >> v1:172.21.15.63:6789/0 conn(0x7f02e413f710 legacy=0x7f02e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 10 monclient: _finish_hunting 0 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 1 monclient: found mon.noname-c 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e40b44c0 con 0x7f02e41430c0 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 10 monclient: _finish_auth 0 2024-03-25T14:26:51.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 10 monclient: _check_auth_tickets 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02dc0019f0 con 0x7f02e41430c0 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02e37fe640 10 monclient: dump: 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:51.111 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:51.112 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:51.112 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:51.112 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:51.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02eab40640 5 monclient: authenticate success, global_id 4540 2024-03-25T14:26:51.112 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.105+0000 7f02eab40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02dc001b50 con 0x7f02e41430c0 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02e37fe640 1 -- 172.21.15.63:0/1556847370 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02dc001fa0 con 0x7f02e41430c0 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 4 set_mon_vals no callback set 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:51.119 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02e37fe640 10 monclient: dump: 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:51.120 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:51.121 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:51.121 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 10 monclient: shutdown 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 1 -- 172.21.15.63:0/1556847370 >> v1:172.21.15.153:6789/0 conn(0x7f02e41430c0 legacy=0x7f02e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 1 -- 172.21.15.63:0/1556847370 shutdown_connections 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 1 -- 172.21.15.63:0/1556847370 >> 172.21.15.63:0/1556847370 conn(0x7f02e40b7ea0 msgr2=0x7f02e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 1 -- 172.21.15.63:0/1556847370 shutdown_connections 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.115+0000 7f02eab40640 1 -- 172.21.15.63:0/1556847370 wait complete. 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 10 monclient: build_initial_monmap 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 10 monclient: monmap: 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:51.124 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 1 librados: starting msgr at 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 1 librados: starting objecter 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 1 Processor -- start 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 1 -- start start 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 1 librados: setting wanted keys 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 1 librados: calling monclient init 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.116+0000 7f02eab40640 10 monclient: init 2024-03-25T14:26:51.125 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 10 monclient(hunting): picked mon.noname-c con 0x7f02e40a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 10 monclient(hunting): picked mon.noname-a con 0x7f02e413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 10 monclient(hunting): picked mon.noname-b con 0x7f02e41430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e4138ad0 con 0x7f02e413f710 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e41d7dc0 con 0x7f02e41430c0 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e41d7f50 con 0x7f02e40a98f0 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:51.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.117+0000 7f02eab40640 10 monclient(hunting): authenticate will time out at 1774.262317s 2024-03-25T14:26:51.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.124+0000 7f02e90dd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f02e41430c0 0x7f02e41d76b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:57598/0 (socket says 172.21.15.63:57598) 2024-03-25T14:26:51.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.124+0000 7f02e90dd640 1 -- 172.21.15.63:0/2230946885 learned_addr learned my addr 172.21.15.63:0/2230946885 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:51.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.125+0000 7f02e17fa640 10 client.?.objecter ms_handle_connect 0x7f02e41430c0 2024-03-25T14:26:51.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.125+0000 7f02e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:51.133 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.129+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471384591 0 0) 0x7f02e41d7dc0 con 0x7f02e41430c0 2024-03-25T14:26:51.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.129+0000 7f02e17fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:51.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.129+0000 7f02e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:51.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.129+0000 7f02e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:51.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.129+0000 7f02e17fa640 10 monclient(hunting): my global_id is 4543 2024-03-25T14:26:51.134 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.129+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c8004eb0 con 0x7f02e41430c0 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.135+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3439339812 0 0) 0x7f02c8004eb0 con 0x7f02e41430c0 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.135+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e41d7dc0 con 0x7f02e41430c0 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.135+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02dc001f20 con 0x7f02e41430c0 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.136+0000 7f02e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.136+0000 7f02e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.136+0000 7f02e17fa640 10 monclient(hunting): dump: 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.140 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.136+0000 7f02e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.137+0000 7f02e17fa640 10 client.?.objecter ms_handle_connect 0x7f02e413f710 2024-03-25T14:26:51.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.137+0000 7f02e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:51.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.138+0000 7f02e17fa640 10 client.?.objecter ms_handle_connect 0x7f02e40a98f0 2024-03-25T14:26:51.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.138+0000 7f02e17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4035112099 0 0) 0x7f02e41d7dc0 con 0x7f02e41430c0 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 >> v1:172.21.15.63:6790/0 conn(0x7f02e40a98f0 legacy=0x7f02e4138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 >> v1:172.21.15.63:6789/0 conn(0x7f02e413f710 legacy=0x7f02e41379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 10 monclient: _finish_hunting 0 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 1 monclient: found mon.b 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e41d80e0 con 0x7f02e41430c0 2024-03-25T14:26:51.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 10 monclient: _finish_auth 0 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 10 monclient: _check_auth_tickets 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 5 monclient: authenticate success, global_id 4543 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 10 monclient: _renew_subs 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e41d8390 con 0x7f02e41430c0 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 10 monclient: _renew_subs 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e41e31a0 con 0x7f02e41430c0 2024-03-25T14:26:51.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.145+0000 7f02eab40640 1 librados: init done 2024-03-25T14:26:51.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.146+0000 7f02eab40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:51.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.146+0000 7f02eab40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:51.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.146+0000 7f02eab40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:51.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.146+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e40627b0 con 0x7f02e41430c0 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.176+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02dc004e10 con 0x7f02e41430c0 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.176+0000 7f02e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.176+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f02dc005120 con 0x7f02e41430c0 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.177+0000 7f02e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.177+0000 7f02e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.177+0000 7f02e17fa640 10 monclient: dump: 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:51.181 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:51.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.177+0000 7f02e2ffd640 4 set_mon_vals no callback set 2024-03-25T14:26:51.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.178+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f02dc01b0f0 con 0x7f02e41430c0 2024-03-25T14:26:51.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.178+0000 7f02e88dc640 10 monclient: get_auth_request con 0x7f02c806dbb0 auth_method 0 2024-03-25T14:26:51.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.179+0000 7f02e17fa640 10 client.4543.objecter ms_handle_connect 0x7f02c806dbb0 2024-03-25T14:26:51.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.181+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f02dc054d40 con 0x7f02e41430c0 2024-03-25T14:26:51.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.181+0000 7f02e17fa640 10 client.4543.objecter ms_dispatch 0x7f02e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:51.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.181+0000 7f02e17fa640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:51.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.181+0000 7f02e17fa640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:51.186 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.181+0000 7f02e17fa640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-25T14:26:51.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.449+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02dc050680 con 0x7f02e41430c0 2024-03-25T14:26:51.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.449+0000 7f02e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:51.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.449+0000 7f02e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:51.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.686+0000 7f02eab40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:51.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.686+0000 7f02eab40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:51.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.686+0000 7f02eab40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:26:51.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.686+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02e4138c60 con 0x7f02e41430c0 2024-03-25T14:26:51.886 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.882+0000 7f02e17fa640 1 -- 172.21.15.63:0/2230946885 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f02dc052780 con 0x7f02e41430c0 2024-03-25T14:26:51.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.882+0000 7f02e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:51.887 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.882+0000 7f02e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:51.887 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:51.887 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:51.888 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.884+0000 7f02eab40640 10 librados: watch_flush enter 2024-03-25T14:26:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 10 librados: watch_flush exit 2024-03-25T14:26:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-25T14:26:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 10 client.4543.objecter successfully canceled tick 2024-03-25T14:26:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 >> v1:172.21.15.153:6800/1688255552 conn(0x7f02c806dbb0 legacy=0x7f02c8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 10 monclient: shutdown 2024-03-25T14:26:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:51.889 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 >> v1:172.21.15.153:6789/0 conn(0x7f02e41430c0 legacy=0x7f02e41d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 shutdown_connections 2024-03-25T14:26:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 >> 172.21.15.63:0/2230946885 conn(0x7f02e40b7ea0 msgr2=0x7f02e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 shutdown_connections 2024-03-25T14:26:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 1 -- 172.21.15.63:0/2230946885 wait complete. 2024-03-25T14:26:51.890 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:51.885+0000 7f02eab40640 1 librados: shutdown 2024-03-25T14:26:51.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:26:58.710 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.824+0000 7fefcb225640 10 monclient: get_monmap_and_config 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.824+0000 7fefcb225640 10 monclient: build_initial_monmap 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.824+0000 7fefcb225640 10 monclient: monmap: 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:26:58.825741+0000 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:26:58.825741+0000 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:26:58.829 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:26:58.830 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:58.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.825+0000 7fefcb225640 1 Processor -- start 2024-03-25T14:26:58.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.825+0000 7fefcb225640 1 -- start start 2024-03-25T14:26:58.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.825+0000 7fefcb225640 10 monclient: init 2024-03-25T14:26:58.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:58.831 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 10 monclient(hunting): picked mon.noname-a con 0x7fefc40a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 10 monclient(hunting): picked mon.noname-b con 0x7fefc413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 10 monclient(hunting): picked mon.noname-c con 0x7fefc41430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc4142520 con 0x7fefc40a98f0 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40b28a0 con 0x7fefc41430c0 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40b2a30 con 0x7fefc413f710 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:58.832 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.826+0000 7fefcb225640 10 monclient(hunting): authenticate will time out at 1781.971369s 2024-03-25T14:26:58.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.835+0000 7fefc8fc1640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fefc40a98f0 0x7fefc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:39020/0 (socket says 172.21.15.63:39020) 2024-03-25T14:26:58.839 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.835+0000 7fefc8fc1640 1 -- 172.21.15.63:0/945329969 learned_addr learned my addr 172.21.15.63:0/945329969 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:58.863 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.858+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313166341 0 0) 0x7fefc4142520 con 0x7fefc40a98f0 2024-03-25T14:26:58.863 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.858+0000 7fefc37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:58.863 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.858+0000 7fefc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:58.863 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.858+0000 7fefc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:58.863 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.858+0000 7fefc37fe640 10 monclient(hunting): my global_id is 4542 2024-03-25T14:26:58.863 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.858+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb4003520 con 0x7fefc40a98f0 2024-03-25T14:26:58.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.860+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187743345 0 0) 0x7fefc40b28a0 con 0x7fefc41430c0 2024-03-25T14:26:58.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.860+0000 7fefc37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:58.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.860+0000 7fefc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:58.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.860+0000 7fefc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:58.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.860+0000 7fefc37fe640 10 monclient(hunting): my global_id is 4546 2024-03-25T14:26:58.865 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.860+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc4142520 con 0x7fefc41430c0 2024-03-25T14:26:58.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.867+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236693915 0 0) 0x7fefc40b2a30 con 0x7fefc413f710 2024-03-25T14:26:58.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.867+0000 7fefc37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:26:58.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.867+0000 7fefc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:58.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.867+0000 7fefc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:58.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.867+0000 7fefc37fe640 10 monclient(hunting): my global_id is 4520 2024-03-25T14:26:58.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.867+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc40b28a0 con 0x7fefc413f710 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786700619 0 0) 0x7fefc4142520 con 0x7fefc41430c0 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 >> v1:172.21.15.63:6790/0 conn(0x7fefc413f710 legacy=0x7fefc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 >> v1:172.21.15.63:6789/0 conn(0x7fefc40a98f0 legacy=0x7fefc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 10 monclient: _finish_hunting 0 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 1 monclient: found mon.noname-c 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc40b44c0 con 0x7fefc41430c0 2024-03-25T14:26:58.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 10 monclient: _finish_auth 0 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 10 monclient: _check_auth_tickets 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fefbc0019f0 con 0x7fefc41430c0 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefc37fe640 10 monclient: dump: 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:58.880 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:58.881 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:58.881 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:58.881 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:58.881 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:58.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefcb225640 5 monclient: authenticate success, global_id 4546 2024-03-25T14:26:58.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.874+0000 7fefcb225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:26:58.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefbc001b50 con 0x7fefc41430c0 2024-03-25T14:26:58.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefc37fe640 1 -- 172.21.15.63:0/945329969 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fefbc001f60 con 0x7fefc41430c0 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefc37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefc37fe640 10 monclient: dump: 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:58.894 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:58.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefcb225640 10 monclient: get_monmap_and_config success 2024-03-25T14:26:58.895 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.889+0000 7fefcb225640 4 set_mon_vals no callback set 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.890+0000 7fefcb225640 10 monclient: shutdown 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.890+0000 7fefcb225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.890+0000 7fefcb225640 1 -- 172.21.15.63:0/945329969 >> v1:172.21.15.153:6789/0 conn(0x7fefc41430c0 legacy=0x7fefc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.890+0000 7fefcb225640 1 -- 172.21.15.63:0/945329969 shutdown_connections 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.890+0000 7fefcb225640 1 -- 172.21.15.63:0/945329969 >> 172.21.15.63:0/945329969 conn(0x7fefc40b7ea0 msgr2=0x7fefc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.890+0000 7fefcb225640 1 -- 172.21.15.63:0/945329969 shutdown_connections 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.890+0000 7fefcb225640 1 -- 172.21.15.63:0/945329969 wait complete. 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 10 monclient: build_initial_monmap 2024-03-25T14:26:58.897 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 10 monclient: monmap: 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 1 librados: starting msgr at 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 1 librados: starting objecter 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 1 Processor -- start 2024-03-25T14:26:58.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 1 -- start start 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 1 librados: setting wanted keys 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 1 librados: calling monclient init 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.891+0000 7fefcb225640 10 monclient: init 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 10 monclient: _reopen_session rank -1 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 10 monclient(hunting): picked mon.noname-a con 0x7fefc40a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 10 monclient(hunting): picked mon.noname-b con 0x7fefc413f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 10 monclient(hunting): picked mon.noname-c con 0x7fefc41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc4137ce0 con 0x7fefc40a98f0 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc41396e0 con 0x7fefc413f710 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc41399c0 con 0x7fefc41430c0 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 10 monclient(hunting): _renew_subs 2024-03-25T14:26:58.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.892+0000 7fefcb225640 10 monclient(hunting): authenticate will time out at 1782.037189s 2024-03-25T14:26:58.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.899+0000 7fefc3fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fefc413f710 0x7fefc4138320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:34082/0 (socket says 172.21.15.63:34082) 2024-03-25T14:26:58.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.899+0000 7fefc3fff640 1 -- 172.21.15.63:0/3307231778 learned_addr learned my addr 172.21.15.63:0/3307231778 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:26:58.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.902+0000 7fefc17fa640 10 client.?.objecter ms_handle_connect 0x7fefc413f710 2024-03-25T14:26:58.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.902+0000 7fefc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:58.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 10 client.?.objecter ms_handle_connect 0x7fefc41430c0 2024-03-25T14:26:58.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:58.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236236418 0 0) 0x7fefc41396e0 con 0x7fefc413f710 2024-03-25T14:26:58.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:58.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:58.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:58.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 10 monclient(hunting): my global_id is 4549 2024-03-25T14:26:58.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.908+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa8004eb0 con 0x7fefc413f710 2024-03-25T14:26:58.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.920+0000 7fefc17fa640 10 client.?.objecter ms_handle_connect 0x7fefc40a98f0 2024-03-25T14:26:58.924 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.920+0000 7fefc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:26:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.925+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476940564 0 0) 0x7fefc41399c0 con 0x7fefc41430c0 2024-03-25T14:26:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.925+0000 7fefc17fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.925+0000 7fefc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.925+0000 7fefc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.925+0000 7fefc17fa640 10 monclient(hunting): my global_id is 4523 2024-03-25T14:26:58.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.925+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc41396e0 con 0x7fefc41430c0 2024-03-25T14:26:58.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.931+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1934447833 0 0) 0x7fefc41396e0 con 0x7fefc41430c0 2024-03-25T14:26:58.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.931+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefc41399c0 con 0x7fefc41430c0 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.932+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fefbc001ea0 con 0x7fefc41430c0 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.932+0000 7fefc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.932+0000 7fefc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.932+0000 7fefc17fa640 10 monclient(hunting): dump: 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:58.937 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:58.938 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:58.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.932+0000 7fefc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:26:58.938 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774903002 0 0) 0x7fefc4137ce0 con 0x7fefc40a98f0 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 10 monclient(hunting): _negotiate 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 10 monclient(hunting): my global_id is 4545 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc41396e0 con 0x7fefc40a98f0 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3240254660 0 0) 0x7fefa8004eb0 con 0x7fefc413f710 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefc4137ce0 con 0x7fefc413f710 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fefb0001a10 con 0x7fefc413f710 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.933+0000 7fefc17fa640 10 monclient(hunting): dump: 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:58.939 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:58.940 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 249756030 0 0) 0x7fefc41399c0 con 0x7fefc41430c0 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 >> v1:172.21.15.153:6789/0 conn(0x7fefc413f710 legacy=0x7fefc4138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 >> v1:172.21.15.63:6789/0 conn(0x7fefc40a98f0 legacy=0x7fefc41da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 10 monclient: _finish_hunting 0 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 1 monclient: found mon.c 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc40b0fb0 con 0x7fefc41430c0 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 10 monclient: _finish_auth 0 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 10 monclient: _check_auth_tickets 2024-03-25T14:26:58.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 5 monclient: authenticate success, global_id 4523 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 10 monclient: _renew_subs 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc40afdc0 con 0x7fefc41430c0 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 10 monclient: _renew_subs 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc40b02f0 con 0x7fefc41430c0 2024-03-25T14:26:58.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.939+0000 7fefcb225640 1 librados: init done 2024-03-25T14:26:58.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.940+0000 7fefcb225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:58.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.940+0000 7fefcb225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:58.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.940+0000 7fefcb225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:58.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:58.940+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc40627b0 con 0x7fefc41430c0 2024-03-25T14:26:59.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.157+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefbc002000 con 0x7fefc41430c0 2024-03-25T14:26:59.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.157+0000 7fefc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:26:59.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.157+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fefbc003dc0 con 0x7fefc41430c0 2024-03-25T14:26:59.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.157+0000 7fefc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.157+0000 7fefc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.157+0000 7fefc17fa640 10 monclient: dump: 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:26:59.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.157+0000 7fefc2ffd640 4 set_mon_vals no callback set 2024-03-25T14:26:59.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.239+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fefbc019d90 con 0x7fefc41430c0 2024-03-25T14:26:59.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.239+0000 7fefc8fc1640 10 monclient: get_auth_request con 0x7fefa806df90 auth_method 0 2024-03-25T14:26:59.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.240+0000 7fefc17fa640 10 client.4523.objecter ms_handle_connect 0x7fefa806df90 2024-03-25T14:26:59.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fefbc0538d0 con 0x7fefc41430c0 2024-03-25T14:26:59.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 10 client.4523.objecter ms_dispatch 0x7fefc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:26:59.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:26:59.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:26:59.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T14:26:59.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefbc04f3e0 con 0x7fefc41430c0 2024-03-25T14:26:59.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:26:59.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.245+0000 7fefc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:26:59.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.473+0000 7fefcb225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:59.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.473+0000 7fefcb225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:59.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.473+0000 7fefcb225640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:26:59.477 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.473+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefc40b71a0 con 0x7fefc41430c0 2024-03-25T14:26:59.509 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.505+0000 7fefc17fa640 1 -- 172.21.15.63:0/3307231778 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fefbc051310 con 0x7fefc41430c0 2024-03-25T14:26:59.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.505+0000 7fefc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:26:59.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.505+0000 7fefc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:26:59.510 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:26:59.510 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 10 librados: watch_flush enter 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 10 librados: watch_flush exit 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 10 client.4523.objecter successfully canceled tick 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 >> v1:172.21.15.153:6800/1688255552 conn(0x7fefa806df90 legacy=0x7fefa8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 10 monclient: shutdown 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 >> v1:172.21.15.63:6790/0 conn(0x7fefc41430c0 legacy=0x7fefc41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 shutdown_connections 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 >> 172.21.15.63:0/3307231778 conn(0x7fefc40b7ea0 msgr2=0x7fefc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 shutdown_connections 2024-03-25T14:26:59.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 1 -- 172.21.15.63:0/3307231778 wait complete. 2024-03-25T14:26:59.514 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:26:59.508+0000 7fefcb225640 1 librados: shutdown 2024-03-25T14:26:59.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:06.331 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.439+0000 7fb1449a8640 10 monclient: get_monmap_and_config 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.439+0000 7fb1449a8640 10 monclient: build_initial_monmap 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.439+0000 7fb1449a8640 10 monclient: monmap: 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:27:06.440729+0000 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:27:06.440729+0000 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:06.444 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:27:06.445 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.440+0000 7fb1449a8640 1 Processor -- start 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.440+0000 7fb1449a8640 1 -- start start 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.440+0000 7fb1449a8640 10 monclient: init 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:06.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb1400a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb14013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb1401430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb140142520 con 0x7fb1401430c0 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1400b28a0 con 0x7fb1400a98f0 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1400b2a30 con 0x7fb14013f710 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:06.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.441+0000 7fb1449a8640 10 monclient(hunting): authenticate will time out at 1789.585958s 2024-03-25T14:27:06.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.452+0000 7fb13e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb1400a98f0 0x7fb1400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:52262/0 (socket says 172.21.15.63:52262) 2024-03-25T14:27:06.456 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.452+0000 7fb13e59c640 1 -- 172.21.15.63:0/1406244037 learned_addr learned my addr 172.21.15.63:0/1406244037 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:06.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.463+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150985709 0 0) 0x7fb1400b28a0 con 0x7fb1400a98f0 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.463+0000 7fb13d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.463+0000 7fb13d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.463+0000 7fb13d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.463+0000 7fb13d59a640 10 monclient(hunting): my global_id is 4552 2024-03-25T14:27:06.468 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.463+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb128003520 con 0x7fb1400a98f0 2024-03-25T14:27:06.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.469+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596182466 0 0) 0x7fb140142520 con 0x7fb1401430c0 2024-03-25T14:27:06.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.469+0000 7fb13d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.469+0000 7fb13d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.469+0000 7fb13d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.469+0000 7fb13d59a640 10 monclient(hunting): my global_id is 4548 2024-03-25T14:27:06.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.469+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1400b28a0 con 0x7fb1401430c0 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 790214337 0 0) 0x7fb128003520 con 0x7fb1400a98f0 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 >> v1:172.21.15.63:6790/0 conn(0x7fb14013f710 legacy=0x7fb1401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 >> v1:172.21.15.63:6789/0 conn(0x7fb1401430c0 legacy=0x7fb14014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 10 monclient: _finish_hunting 0 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 1 monclient: found mon.noname-c 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1400b44c0 con 0x7fb1400a98f0 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 10 monclient: _finish_auth 0 2024-03-25T14:27:06.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 10 monclient: _check_auth_tickets 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb130001690 con 0x7fb1400a98f0 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb1449a8640 5 monclient: authenticate success, global_id 4552 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb1449a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb13d59a640 10 monclient: dump: 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.482 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:06.483 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:06.483 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:06.483 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:06.483 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.483 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.476+0000 7fb1449a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:06.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1300017f0 con 0x7fb1400a98f0 2024-03-25T14:27:06.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb13d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:06.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb13d59a640 1 -- 172.21.15.63:0/1406244037 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb130001c00 con 0x7fb1400a98f0 2024-03-25T14:27:06.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb13d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb13d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:06.486 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb13d59a640 10 monclient: dump: 2024-03-25T14:27:06.486 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb1449a8640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:06.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb1449a8640 4 set_mon_vals no callback set 2024-03-25T14:27:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb1449a8640 10 monclient: shutdown 2024-03-25T14:27:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb1449a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.482+0000 7fb1449a8640 1 -- 172.21.15.63:0/1406244037 >> v1:172.21.15.153:6789/0 conn(0x7fb1400a98f0 legacy=0x7fb1400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.483+0000 7fb1449a8640 1 -- 172.21.15.63:0/1406244037 shutdown_connections 2024-03-25T14:27:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.483+0000 7fb1449a8640 1 -- 172.21.15.63:0/1406244037 >> 172.21.15.63:0/1406244037 conn(0x7fb1400b7ea0 msgr2=0x7fb1400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:06.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.483+0000 7fb1449a8640 1 -- 172.21.15.63:0/1406244037 shutdown_connections 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.483+0000 7fb1449a8640 1 -- 172.21.15.63:0/1406244037 wait complete. 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.483+0000 7fb1449a8640 10 monclient: build_initial_monmap 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.483+0000 7fb1449a8640 10 monclient: monmap: 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:27:06.490 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 librados: starting msgr at 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 librados: starting objecter 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 Processor -- start 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 -- start start 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 librados: setting wanted keys 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 librados: calling monclient init 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient: init 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb1400a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb14013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:06.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb1401430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb140138250 con 0x7fb14013f710 2024-03-25T14:27:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb140137d50 con 0x7fb1400a98f0 2024-03-25T14:27:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb140137ee0 con 0x7fb1401430c0 2024-03-25T14:27:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:06.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.484+0000 7fb1449a8640 10 monclient(hunting): authenticate will time out at 1789.629465s 2024-03-25T14:27:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.489+0000 7fb13e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb1400a98f0 0x7fb140137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:52274/0 (socket says 172.21.15.63:52274) 2024-03-25T14:27:06.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.489+0000 7fb13e59c640 1 -- 172.21.15.63:0/2852179119 learned_addr learned my addr 172.21.15.63:0/2852179119 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:06.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.493+0000 7fb12effd640 10 client.?.objecter ms_handle_connect 0x7fb1401430c0 2024-03-25T14:27:06.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.493+0000 7fb12effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:06.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.493+0000 7fb12effd640 10 client.?.objecter ms_handle_connect 0x7fb1400a98f0 2024-03-25T14:27:06.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.493+0000 7fb12effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:06.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.500+0000 7fb12effd640 10 client.?.objecter ms_handle_connect 0x7fb14013f710 2024-03-25T14:27:06.504 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.500+0000 7fb12effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:06.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.507+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950192589 0 0) 0x7fb140137d50 con 0x7fb1400a98f0 2024-03-25T14:27:06.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.507+0000 7fb12effd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.507+0000 7fb12effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.507+0000 7fb12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.507+0000 7fb12effd640 10 monclient(hunting): my global_id is 4555 2024-03-25T14:27:06.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.507+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb120004eb0 con 0x7fb1400a98f0 2024-03-25T14:27:06.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.512+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802793156 0 0) 0x7fb140137ee0 con 0x7fb1401430c0 2024-03-25T14:27:06.516 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.512+0000 7fb12effd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.512+0000 7fb12effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.512+0000 7fb12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.512+0000 7fb12effd640 10 monclient(hunting): my global_id is 4529 2024-03-25T14:27:06.517 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.512+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb140137d50 con 0x7fb1401430c0 2024-03-25T14:27:06.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.513+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334724689 0 0) 0x7fb140138250 con 0x7fb14013f710 2024-03-25T14:27:06.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.513+0000 7fb12effd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:06.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.513+0000 7fb12effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:06.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.513+0000 7fb12effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:06.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.513+0000 7fb12effd640 10 monclient(hunting): my global_id is 4551 2024-03-25T14:27:06.518 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.513+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb140137ee0 con 0x7fb14013f710 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377984816 0 0) 0x7fb140137ee0 con 0x7fb14013f710 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb140138250 con 0x7fb14013f710 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb134001860 con 0x7fb14013f710 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 10 monclient(hunting): dump: 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.539 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3994722020 0 0) 0x7fb120004eb0 con 0x7fb1400a98f0 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb140137ee0 con 0x7fb1400a98f0 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb130001c00 con 0x7fb1400a98f0 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.534+0000 7fb12effd640 10 monclient(hunting): dump: 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:06.540 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:06.541 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.538+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929799819 0 0) 0x7fb140137d50 con 0x7fb1401430c0 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.538+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb120004eb0 con 0x7fb1401430c0 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.538+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb138001b90 con 0x7fb1401430c0 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.538+0000 7fb12effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.538+0000 7fb12effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.538+0000 7fb12effd640 10 monclient(hunting): dump: 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:06.543 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:06.544 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:06.544 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3575394994 0 0) 0x7fb140137ee0 con 0x7fb1400a98f0 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 >> v1:172.21.15.63:6790/0 conn(0x7fb1401430c0 legacy=0x7fb1401d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 >> v1:172.21.15.63:6789/0 conn(0x7fb14013f710 legacy=0x7fb1401d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 10 monclient: _finish_hunting 0 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 1 monclient: found mon.b 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1400b7730 con 0x7fb1400a98f0 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 10 monclient: _finish_auth 0 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 10 monclient: _check_auth_tickets 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb12effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:06.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 5 monclient: authenticate success, global_id 4555 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 10 monclient: _renew_subs 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1400b7920 con 0x7fb1400a98f0 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 10 monclient: _renew_subs 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1401e33d0 con 0x7fb1400a98f0 2024-03-25T14:27:06.552 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.546+0000 7fb1449a8640 1 librados: init done 2024-03-25T14:27:06.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.547+0000 7fb1449a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:06.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.547+0000 7fb1449a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:06.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.547+0000 7fb1449a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:06.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.547+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1400627b0 con 0x7fb1400a98f0 2024-03-25T14:27:06.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.837+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb130004a90 con 0x7fb1400a98f0 2024-03-25T14:27:06.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.837+0000 7fb12effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:06.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.837+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fb130004e60 con 0x7fb1400a98f0 2024-03-25T14:27:06.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.837+0000 7fb12effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.837+0000 7fb12effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.837+0000 7fb12effd640 10 monclient: dump: 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:06.842 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.837+0000 7fb13cd99640 4 set_mon_vals no callback set 2024-03-25T14:27:06.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.838+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fb130005080 con 0x7fb1400a98f0 2024-03-25T14:27:06.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.838+0000 7fb13dd9b640 10 monclient: get_auth_request con 0x7fb12006e160 auth_method 0 2024-03-25T14:27:06.843 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.839+0000 7fb12effd640 10 client.4555.objecter ms_handle_connect 0x7fb12006e160 2024-03-25T14:27:06.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fb130054930 con 0x7fb1400a98f0 2024-03-25T14:27:06.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 10 client.4555.objecter ms_dispatch 0x7fb140000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:06.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:06.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:06.849 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T14:27:06.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb130050440 con 0x7fb1400a98f0 2024-03-25T14:27:06.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:06.850 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:06.845+0000 7fb12effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:07.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.095+0000 7fb1449a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:07.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.095+0000 7fb1449a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:07.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.095+0000 7fb1449a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:07.100 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.095+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb14013df50 con 0x7fb1400a98f0 2024-03-25T14:27:07.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.144+0000 7fb12effd640 1 -- 172.21.15.63:0/2852179119 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fb130052370 con 0x7fb1400a98f0 2024-03-25T14:27:07.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.144+0000 7fb12effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:07.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.144+0000 7fb12effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:07.149 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:27:07.149 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:07.151 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.146+0000 7fb1449a8640 10 librados: watch_flush enter 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 10 librados: watch_flush exit 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 10 client.4555.objecter successfully canceled tick 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 >> v1:172.21.15.153:6800/1688255552 conn(0x7fb12006e160 legacy=0x7fb120070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 10 monclient: shutdown 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 >> v1:172.21.15.153:6789/0 conn(0x7fb1400a98f0 legacy=0x7fb140137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 shutdown_connections 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 >> 172.21.15.63:0/2852179119 conn(0x7fb1400b7ea0 msgr2=0x7fb140074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 shutdown_connections 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 1 -- 172.21.15.63:0/2852179119 wait complete. 2024-03-25T14:27:07.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:07.147+0000 7fb1449a8640 1 librados: shutdown 2024-03-25T14:27:07.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:07.295 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:45.807 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.300 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:45.812 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.335 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:07:45.847 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.357 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:45.817 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.376 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:45.835 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.384 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:45.896 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.395 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:45.854 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.400 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:45.912 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:07.425 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:07:45.937 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:13.969 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.079+0000 7fa6cd913640 10 monclient: get_monmap_and_config 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.079+0000 7fa6cd913640 10 monclient: build_initial_monmap 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.079+0000 7fa6cd913640 10 monclient: monmap: 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:27:14.080738+0000 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:27:14.080738+0000 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:14.084 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:27:14.085 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:27:14.085 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:14.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.080+0000 7fa6cd913640 1 Processor -- start 2024-03-25T14:27:14.085 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.080+0000 7fa6cd913640 1 -- start start 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.080+0000 7fa6cd913640 10 monclient: init 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c813fc60 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c8070570 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c80727a0 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c813c9f0 con 0x7fa6c8070570 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c806fac0 con 0x7fa6c813fc60 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80715d0 con 0x7fa6c80727a0 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:14.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.081+0000 7fa6cd913640 10 monclient(hunting): authenticate will time out at 1797.226304s 2024-03-25T14:27:14.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.088+0000 7fa6c6ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa6c813fc60 0x7fa6c806f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:53092/0 (socket says 172.21.15.63:53092) 2024-03-25T14:27:14.092 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.088+0000 7fa6c6ffd640 1 -- 172.21.15.63:0/1401751199 learned_addr learned my addr 172.21.15.63:0/1401751199 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:14.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.102+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528835168 0 0) 0x7fa6c806fac0 con 0x7fa6c813fc60 2024-03-25T14:27:14.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.103+0000 7fa6c67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.103+0000 7fa6c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.103+0000 7fa6c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.103+0000 7fa6c67fc640 10 monclient(hunting): my global_id is 4558 2024-03-25T14:27:14.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.103+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4003520 con 0x7fa6c813fc60 2024-03-25T14:27:14.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.111+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295727203 0 0) 0x7fa6c813c9f0 con 0x7fa6c8070570 2024-03-25T14:27:14.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.111+0000 7fa6c67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.111+0000 7fa6c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.111+0000 7fa6c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.111+0000 7fa6c67fc640 10 monclient(hunting): my global_id is 4554 2024-03-25T14:27:14.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.111+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c806fac0 con 0x7fa6c8070570 2024-03-25T14:27:14.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.118+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950440481 0 0) 0x7fa6c80715d0 con 0x7fa6c80727a0 2024-03-25T14:27:14.122 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.118+0000 7fa6c67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.118+0000 7fa6c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.118+0000 7fa6c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.118+0000 7fa6c67fc640 10 monclient(hunting): my global_id is 4532 2024-03-25T14:27:14.123 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.118+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c813c9f0 con 0x7fa6c80727a0 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665924726 0 0) 0x7fa6b4003520 con 0x7fa6c813fc60 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 >> v1:172.21.15.63:6790/0 conn(0x7fa6c80727a0 legacy=0x7fa6c814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 >> v1:172.21.15.63:6789/0 conn(0x7fa6c8070570 legacy=0x7fa6c8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 1 monclient: found mon.noname-c 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c80718b0 con 0x7fa6c813fc60 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:14.128 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa6b8001690 con 0x7fa6c813fc60 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6cd913640 5 monclient: authenticate success, global_id 4558 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6c67fc640 10 monclient: dump: 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.129 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:14.130 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:14.130 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:14.130 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:14.130 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:14.130 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.123+0000 7fa6cd913640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:14.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6b80017f0 con 0x7fa6c813fc60 2024-03-25T14:27:14.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:14.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6c67fc640 1 -- 172.21.15.63:0/1401751199 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa6b8001c00 con 0x7fa6c813fc60 2024-03-25T14:27:14.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:14.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 4 set_mon_vals no callback set 2024-03-25T14:27:14.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.135 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6c67fc640 10 monclient: dump: 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:14.136 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 10 monclient: shutdown 2024-03-25T14:27:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 1 -- 172.21.15.63:0/1401751199 >> v1:172.21.15.153:6789/0 conn(0x7fa6c813fc60 legacy=0x7fa6c806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.138 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 1 -- 172.21.15.63:0/1401751199 shutdown_connections 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 1 -- 172.21.15.63:0/1401751199 >> 172.21.15.63:0/1401751199 conn(0x7fa6c80b4a80 msgr2=0x7fa6c80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 1 -- 172.21.15.63:0/1401751199 shutdown_connections 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.131+0000 7fa6cd913640 1 -- 172.21.15.63:0/1401751199 wait complete. 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 10 monclient: build_initial_monmap 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 10 monclient: monmap: 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:14.139 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 1 librados: starting msgr at 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 1 librados: starting objecter 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 1 Processor -- start 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 1 -- start start 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 1 librados: setting wanted keys 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 1 librados: calling monclient init 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.132+0000 7fa6cd913640 10 monclient: init 2024-03-25T14:27:14.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c8070570 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c80727a0 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c813fc60 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c81d7f60 con 0x7fa6c8070570 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c81cbdc0 con 0x7fa6c813fc60 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c81cc0a0 con 0x7fa6c80727a0 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:14.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.133+0000 7fa6cd913640 10 monclient(hunting): authenticate will time out at 1797.278084s 2024-03-25T14:27:14.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.138+0000 7fa6c77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa6c813fc60 0x7fa6c81e2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:53116/0 (socket says 172.21.15.63:53116) 2024-03-25T14:27:14.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.138+0000 7fa6c77fe640 1 -- 172.21.15.63:0/3917090073 learned_addr learned my addr 172.21.15.63:0/3917090073 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:14.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.139+0000 7fa6c4ff9640 10 client.?.objecter ms_handle_connect 0x7fa6c813fc60 2024-03-25T14:27:14.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.139+0000 7fa6c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:14.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.143+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599965642 0 0) 0x7fa6c81cbdc0 con 0x7fa6c813fc60 2024-03-25T14:27:14.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.144+0000 7fa6c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.144+0000 7fa6c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.144+0000 7fa6c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.144+0000 7fa6c4ff9640 10 monclient(hunting): my global_id is 4561 2024-03-25T14:27:14.148 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.144+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a8004eb0 con 0x7fa6c813fc60 2024-03-25T14:27:14.152 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.148+0000 7fa6c4ff9640 10 client.?.objecter ms_handle_connect 0x7fa6c8070570 2024-03-25T14:27:14.153 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.148+0000 7fa6c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 10 client.?.objecter ms_handle_connect 0x7fa6c80727a0 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2672639216 0 0) 0x7fa6a8004eb0 con 0x7fa6c813fc60 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6c81cbdc0 con 0x7fa6c813fc60 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa6c0001cd0 con 0x7fa6c813fc60 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 10 monclient(hunting): dump: 2024-03-25T14:27:14.155 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:14.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.150+0000 7fa6c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154468785 0 0) 0x7fa6c81d7f60 con 0x7fa6c8070570 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): my global_id is 4557 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a8004eb0 con 0x7fa6c8070570 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219003982 0 0) 0x7fa6c81cc0a0 con 0x7fa6c80727a0 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:14.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 10 monclient(hunting): my global_id is 4535 2024-03-25T14:27:14.181 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.175+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c81d7f60 con 0x7fa6c80727a0 2024-03-25T14:27:14.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3420242428 0 0) 0x7fa6c81cbdc0 con 0x7fa6c813fc60 2024-03-25T14:27:14.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 >> v1:172.21.15.63:6790/0 conn(0x7fa6c80727a0 legacy=0x7fa6c81c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 >> v1:172.21.15.63:6789/0 conn(0x7fa6c8070570 legacy=0x7fa6c81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 1 monclient: found mon.b 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c81d83f0 con 0x7fa6c813fc60 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 10 monclient: _finish_auth 0 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 5 monclient: authenticate success, global_id 4561 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 10 monclient: _renew_subs 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c81c8a80 con 0x7fa6c813fc60 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 10 monclient: _renew_subs 2024-03-25T14:27:14.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:14.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c8142750 con 0x7fa6c813fc60 2024-03-25T14:27:14.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.183+0000 7fa6cd913640 1 librados: init done 2024-03-25T14:27:14.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.184+0000 7fa6cd913640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:14.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.184+0000 7fa6cd913640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:14.190 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.184+0000 7fa6cd913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:14.191 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.184+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c80627b0 con 0x7fa6c813fc60 2024-03-25T14:27:14.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.468+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6c0003970 con 0x7fa6c813fc60 2024-03-25T14:27:14.472 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.468+0000 7fa6c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.468+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa6c0003dc0 con 0x7fa6c813fc60 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.468+0000 7fa6c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.468+0000 7fa6c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.468+0000 7fa6c4ff9640 10 monclient: dump: 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:14.473 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:14.474 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:14.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.468+0000 7fa6c5ffb640 4 set_mon_vals no callback set 2024-03-25T14:27:14.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.469+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fa6c0003fe0 con 0x7fa6c813fc60 2024-03-25T14:27:14.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.469+0000 7fa6c6ffd640 10 monclient: get_auth_request con 0x7fa6a806de80 auth_method 0 2024-03-25T14:27:14.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.469+0000 7fa6c4ff9640 10 client.4561.objecter ms_handle_connect 0x7fa6a806de80 2024-03-25T14:27:14.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fa6c0054b10 con 0x7fa6c813fc60 2024-03-25T14:27:14.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 10 client.4561.objecter ms_dispatch 0x7fa6c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:14.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:14.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:14.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T14:27:14.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6c0050620 con 0x7fa6c813fc60 2024-03-25T14:27:14.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:14.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.474+0000 7fa6c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:14.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.723+0000 7fa6cd913640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:14.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.723+0000 7fa6cd913640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:14.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.723+0000 7fa6cd913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:14.728 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.723+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6c80af160 con 0x7fa6c813fc60 2024-03-25T14:27:14.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.764+0000 7fa6c4ff9640 1 -- 172.21.15.63:0/3917090073 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fa6c0052550 con 0x7fa6c813fc60 2024-03-25T14:27:14.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.764+0000 7fa6c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:14.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.764+0000 7fa6c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:14.769 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:27:14.769 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.766+0000 7fa6cd913640 10 librados: watch_flush enter 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 10 librados: watch_flush exit 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 10 client.4561.objecter successfully canceled tick 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 >> v1:172.21.15.153:6800/1688255552 conn(0x7fa6a806de80 legacy=0x7fa6a8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 10 monclient: shutdown 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 >> v1:172.21.15.153:6789/0 conn(0x7fa6c813fc60 legacy=0x7fa6c81e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 shutdown_connections 2024-03-25T14:27:14.772 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 >> 172.21.15.63:0/3917090073 conn(0x7fa6c80b4a80 msgr2=0x7fa6c80afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 shutdown_connections 2024-03-25T14:27:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 1 -- 172.21.15.63:0/3917090073 wait complete. 2024-03-25T14:27:14.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:14.767+0000 7fa6cd913640 1 librados: shutdown 2024-03-25T14:27:14.782 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:21.589 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:21.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.699+0000 7f1c63fff640 10 monclient: get_monmap_and_config 2024-03-25T14:27:21.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.699+0000 7f1c63fff640 10 monclient: build_initial_monmap 2024-03-25T14:27:21.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.699+0000 7f1c63fff640 10 monclient: monmap: 2024-03-25T14:27:21.704 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:21.704 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:27:21.700891+0000 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:27:21.700891+0000 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:27:21.705 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:21.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.700+0000 7f1c63fff640 1 Processor -- start 2024-03-25T14:27:21.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.700+0000 7f1c63fff640 1 -- start start 2024-03-25T14:27:21.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.700+0000 7f1c63fff640 10 monclient: init 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1c640b48e0 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6413fc60 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1c640b3f90 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640a9d10 con 0x7f1c640b48e0 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640ab000 con 0x7f1c640b3f90 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640a9680 con 0x7f1c6413fc60 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:21.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.701+0000 7f1c63fff640 10 monclient(hunting): authenticate will time out at 1804.846140s 2024-03-25T14:27:21.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.710+0000 7f1c637fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1c640b3f90 0x7f1c6414dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36822/0 (socket says 172.21.15.63:36822) 2024-03-25T14:27:21.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.710+0000 7f1c637fe640 1 -- 172.21.15.63:0/4215739852 learned_addr learned my addr 172.21.15.63:0/4215739852 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:21.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929040571 0 0) 0x7f1c640a9680 con 0x7f1c6413fc60 2024-03-25T14:27:21.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): my global_id is 4538 2024-03-25T14:27:21.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c003520 con 0x7f1c6413fc60 2024-03-25T14:27:21.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662541104 0 0) 0x7f1c640ab000 con 0x7f1c640b3f90 2024-03-25T14:27:21.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 10 monclient(hunting): my global_id is 4564 2024-03-25T14:27:21.727 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.721+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c640a9680 con 0x7f1c640b3f90 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 937407159 0 0) 0x7f1c640a9680 con 0x7f1c640b3f90 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 >> v1:172.21.15.63:6790/0 conn(0x7f1c6413fc60 legacy=0x7f1c64145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 >> v1:172.21.15.63:6789/0 conn(0x7f1c640b48e0 legacy=0x7f1c6413f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 1 monclient: found mon.noname-c 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:27:21.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640a9810 con 0x7f1c640b3f90 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 10 monclient: _finish_auth 0 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c5c0019f0 con 0x7f1c640b3f90 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c63fff640 5 monclient: authenticate success, global_id 4564 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c63fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:21.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c61ffb640 10 monclient: dump: 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:21.743 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.736+0000 7f1c63fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:21.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c5c001b50 con 0x7f1c640b3f90 2024-03-25T14:27:21.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:21.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c61ffb640 1 -- 172.21.15.63:0/4215739852 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c5c001f20 con 0x7f1c640b3f90 2024-03-25T14:27:21.744 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c63fff640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c63fff640 4 set_mon_vals no callback set 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c61ffb640 10 monclient: dump: 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:21.745 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:21.746 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:21.746 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:21.746 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:21.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c63fff640 10 monclient: shutdown 2024-03-25T14:27:21.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c63fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:21.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.740+0000 7f1c63fff640 1 -- 172.21.15.63:0/4215739852 >> v1:172.21.15.153:6789/0 conn(0x7f1c640b3f90 legacy=0x7f1c6414dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.741+0000 7f1c63fff640 1 -- 172.21.15.63:0/4215739852 shutdown_connections 2024-03-25T14:27:21.748 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.741+0000 7f1c63fff640 1 -- 172.21.15.63:0/4215739852 >> 172.21.15.63:0/4215739852 conn(0x7f1c640615c0 msgr2=0x7f1c640619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.741+0000 7f1c63fff640 1 -- 172.21.15.63:0/4215739852 shutdown_connections 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.741+0000 7f1c63fff640 1 -- 172.21.15.63:0/4215739852 wait complete. 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.741+0000 7f1c63fff640 10 monclient: build_initial_monmap 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.741+0000 7f1c63fff640 10 monclient: monmap: 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:27:21.749 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.742+0000 7f1c63fff640 1 librados: starting msgr at 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.742+0000 7f1c63fff640 1 librados: starting objecter 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.742+0000 7f1c63fff640 1 Processor -- start 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.742+0000 7f1c63fff640 1 -- start start 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.742+0000 7f1c63fff640 1 librados: setting wanted keys 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.742+0000 7f1c63fff640 1 librados: calling monclient init 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.742+0000 7f1c63fff640 10 monclient: init 2024-03-25T14:27:21.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1c640b3f90 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1c640b48e0 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1c6413fc60 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640ad170 con 0x7f1c640b48e0 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640ac2e0 con 0x7f1c640b3f90 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640ab720 con 0x7f1c6413fc60 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:21.751 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.743+0000 7f1c63fff640 10 monclient(hunting): authenticate will time out at 1804.888356s 2024-03-25T14:27:21.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.751+0000 7f1c62ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1c640b3f90 0x7f1c641d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36842/0 (socket says 172.21.15.63:36842) 2024-03-25T14:27:21.755 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.751+0000 7f1c62ffd640 1 -- 172.21.15.63:0/2348569353 learned_addr learned my addr 172.21.15.63:0/2348569353 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:21.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.752+0000 7f1c4b7fe640 10 client.?.objecter ms_handle_connect 0x7f1c640b3f90 2024-03-25T14:27:21.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.752+0000 7f1c4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:21.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.760+0000 7f1c4b7fe640 10 client.?.objecter ms_handle_connect 0x7f1c6413fc60 2024-03-25T14:27:21.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.760+0000 7f1c4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:21.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.761+0000 7f1c4b7fe640 10 client.?.objecter ms_handle_connect 0x7f1c640b48e0 2024-03-25T14:27:21.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.761+0000 7f1c4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889885908 0 0) 0x7f1c640ac2e0 con 0x7f1c640b3f90 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): my global_id is 4567 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c50004eb0 con 0x7f1c640b3f90 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750801104 0 0) 0x7f1c640ad170 con 0x7f1c640b48e0 2024-03-25T14:27:21.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 10 monclient(hunting): my global_id is 4563 2024-03-25T14:27:21.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.765+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c640ac2e0 con 0x7f1c640b48e0 2024-03-25T14:27:21.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.778+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509110589 0 0) 0x7f1c640ab720 con 0x7f1c6413fc60 2024-03-25T14:27:21.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.778+0000 7f1c4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:21.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.778+0000 7f1c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:21.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.778+0000 7f1c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:21.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.778+0000 7f1c4b7fe640 10 monclient(hunting): my global_id is 4541 2024-03-25T14:27:21.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.778+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c640ad170 con 0x7f1c6413fc60 2024-03-25T14:27:21.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.781+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329671502 0 0) 0x7f1c50004eb0 con 0x7f1c640b3f90 2024-03-25T14:27:21.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.781+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c640ab720 con 0x7f1c640b3f90 2024-03-25T14:27:21.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.781+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c54001a30 con 0x7f1c640b3f90 2024-03-25T14:27:21.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.781+0000 7f1c4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:21.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.781+0000 7f1c4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:21.786 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.781+0000 7f1c4b7fe640 10 monclient(hunting): dump: 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:21.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.781+0000 7f1c4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:21.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1470906211 0 0) 0x7f1c640ab720 con 0x7f1c640b3f90 2024-03-25T14:27:21.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 >> v1:172.21.15.63:6790/0 conn(0x7f1c6413fc60 legacy=0x7f1c641e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 >> v1:172.21.15.63:6789/0 conn(0x7f1c640b48e0 legacy=0x7f1c64137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 1 monclient: found mon.b 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c641d7230 con 0x7f1c640b3f90 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 5 monclient: authenticate success, global_id 4567 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 10 monclient: _renew_subs 2024-03-25T14:27:21.791 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:21.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c64140070 con 0x7f1c640b3f90 2024-03-25T14:27:21.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:21.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 10 monclient: _renew_subs 2024-03-25T14:27:21.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:21.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c641d8040 con 0x7f1c640b3f90 2024-03-25T14:27:21.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.786+0000 7f1c63fff640 1 librados: init done 2024-03-25T14:27:21.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.787+0000 7f1c63fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:21.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.787+0000 7f1c63fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:21.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.787+0000 7f1c63fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:21.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:21.787+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c640627b0 con 0x7f1c640b3f90 2024-03-25T14:27:22.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.077+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c54001c70 con 0x7f1c640b3f90 2024-03-25T14:27:22.081 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.077+0000 7f1c4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.077+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1c54003a60 con 0x7f1c640b3f90 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.077+0000 7f1c4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.077+0000 7f1c4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.077+0000 7f1c4b7fe640 10 monclient: dump: 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:22.082 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:22.083 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:22.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.077+0000 7f1c617fa640 4 set_mon_vals no callback set 2024-03-25T14:27:22.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.078+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f1c54003c40 con 0x7f1c640b3f90 2024-03-25T14:27:22.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.078+0000 7f1c627fc640 10 monclient: get_auth_request con 0x7f1c5006ddf0 auth_method 0 2024-03-25T14:27:22.083 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.079+0000 7f1c4b7fe640 10 client.4567.objecter ms_handle_connect 0x7f1c5006ddf0 2024-03-25T14:27:22.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.082+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f1c540548b0 con 0x7f1c640b3f90 2024-03-25T14:27:22.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.082+0000 7f1c4b7fe640 10 client.4567.objecter ms_dispatch 0x7f1c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:22.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.082+0000 7f1c4b7fe640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:22.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.082+0000 7f1c4b7fe640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:22.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.083+0000 7f1c4b7fe640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T14:27:22.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.083+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c540503c0 con 0x7f1c640b3f90 2024-03-25T14:27:22.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.083+0000 7f1c4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:22.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.083+0000 7f1c4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:22.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.349+0000 7f1c63fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:22.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.349+0000 7f1c63fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:22.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.349+0000 7f1c63fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:27:22.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.349+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c641d73c0 con 0x7f1c640b3f90 2024-03-25T14:27:22.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.370+0000 7f1c4b7fe640 1 -- 172.21.15.63:0/2348569353 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f1c540522f0 con 0x7f1c640b3f90 2024-03-25T14:27:22.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.370+0000 7f1c4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:22.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.370+0000 7f1c4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:22.375 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:27:22.375 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 10 librados: watch_flush enter 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 10 librados: watch_flush exit 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 10 client.4567.objecter successfully canceled tick 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 >> v1:172.21.15.153:6800/1688255552 conn(0x7f1c5006ddf0 legacy=0x7f1c500702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 10 monclient: shutdown 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 >> v1:172.21.15.153:6789/0 conn(0x7f1c640b3f90 legacy=0x7f1c641d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 shutdown_connections 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 >> 172.21.15.63:0/2348569353 conn(0x7f1c640615c0 msgr2=0x7f1c640b5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:22.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 shutdown_connections 2024-03-25T14:27:22.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 1 -- 172.21.15.63:0/2348569353 wait complete. 2024-03-25T14:27:22.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:22.372+0000 7f1c63fff640 1 librados: shutdown 2024-03-25T14:27:22.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:27.354 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:05.866 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.357 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:05.870 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.394 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:05.906 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.416 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:05.928 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.422 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:05.881 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.423 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:05.935 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.428 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:05.888 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.449 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:05.962 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:27.461 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:05.921 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:29.195 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.306+0000 7f0010417640 10 monclient: get_monmap_and_config 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.306+0000 7f0010417640 10 monclient: build_initial_monmap 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.306+0000 7f0010417640 10 monclient: monmap: 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:27:29.307891+0000 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:27:29.307891+0000 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:29.311 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:27:29.312 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.307+0000 7f0010417640 1 Processor -- start 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.307+0000 7f0010417640 1 -- start start 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.307+0000 7f0010417640 10 monclient: init 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 10 monclient(hunting): picked mon.noname-a con 0x7f00080b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 10 monclient(hunting): picked mon.noname-b con 0x7f00080b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 10 monclient(hunting): picked mon.noname-c con 0x7f000814be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:29.313 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00080af940 con 0x7f00080b4d60 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00080b0a70 con 0x7f000814be90 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00080b0d50 con 0x7f00080b1730 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:29.314 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.308+0000 7f0010417640 10 monclient(hunting): authenticate will time out at 1812.452897s 2024-03-25T14:27:29.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.328+0000 7f000e9b4640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f000814be90 0x7f000814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:41320/0 (socket says 172.21.15.63:41320) 2024-03-25T14:27:29.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.328+0000 7f000e9b4640 1 -- 172.21.15.63:0/1797433059 learned_addr learned my addr 172.21.15.63:0/1797433059 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:29.374 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.370+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600621319 0 0) 0x7f00080b0d50 con 0x7f00080b1730 2024-03-25T14:27:29.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.370+0000 7f000d1b1640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.370+0000 7f000d1b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.370+0000 7f000d1b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.370+0000 7f000d1b1640 10 monclient(hunting): my global_id is 4544 2024-03-25T14:27:29.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.370+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff4003520 con 0x7f00080b1730 2024-03-25T14:27:29.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.373+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013374004 0 0) 0x7f00080b0a70 con 0x7f000814be90 2024-03-25T14:27:29.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.373+0000 7f000d1b1640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.373+0000 7f000d1b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.373+0000 7f000d1b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.373+0000 7f000d1b1640 10 monclient(hunting): my global_id is 4570 2024-03-25T14:27:29.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.373+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00080b0d50 con 0x7f000814be90 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734179348 0 0) 0x7f00080af940 con 0x7f00080b4d60 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient(hunting): my global_id is 4566 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00080b0a70 con 0x7f00080b4d60 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921605987 0 0) 0x7efff4003520 con 0x7f00080b1730 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 >> v1:172.21.15.153:6789/0 conn(0x7f000814be90 legacy=0x7f000814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 >> v1:172.21.15.63:6789/0 conn(0x7f00080b4d60 legacy=0x7f000813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient: _finish_hunting 0 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 monclient: found mon.noname-b 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00080b1030 con 0x7f00080b1730 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient: _finish_auth 0 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient: _check_auth_tickets 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7efff8001690 con 0x7f00080b1730 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f000d1b1640 10 monclient: dump: 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:29.387 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f0010417640 5 monclient: authenticate success, global_id 4544 2024-03-25T14:27:29.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.381+0000 7f0010417640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:29.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efff80017f0 con 0x7f00080b1730 2024-03-25T14:27:29.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f000d1b1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:29.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f000d1b1640 1 -- 172.21.15.63:0/1797433059 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7efff8001c00 con 0x7f00080b1730 2024-03-25T14:27:29.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f000d1b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f000d1b1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f000d1b1640 10 monclient: dump: 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:29.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f0010417640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:29.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.394+0000 7f0010417640 4 set_mon_vals no callback set 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.395+0000 7f0010417640 10 monclient: shutdown 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.395+0000 7f0010417640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.395+0000 7f0010417640 1 -- 172.21.15.63:0/1797433059 >> v1:172.21.15.63:6790/0 conn(0x7f00080b1730 legacy=0x7f00080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.395+0000 7f0010417640 1 -- 172.21.15.63:0/1797433059 shutdown_connections 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.395+0000 7f0010417640 1 -- 172.21.15.63:0/1797433059 >> 172.21.15.63:0/1797433059 conn(0x7f00080617a0 msgr2=0x7f0008061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.395+0000 7f0010417640 1 -- 172.21.15.63:0/1797433059 shutdown_connections 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.395+0000 7f0010417640 1 -- 172.21.15.63:0/1797433059 wait complete. 2024-03-25T14:27:29.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 10 monclient: build_initial_monmap 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 10 monclient: monmap: 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 1 librados: starting msgr at 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 1 librados: starting objecter 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 1 Processor -- start 2024-03-25T14:27:29.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 1 -- start start 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 1 librados: setting wanted keys 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 1 librados: calling monclient init 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.396+0000 7f0010417640 10 monclient: init 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 10 monclient(hunting): picked mon.noname-a con 0x7f00080b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 10 monclient(hunting): picked mon.noname-b con 0x7f00080b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 10 monclient(hunting): picked mon.noname-c con 0x7f000814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0008140890 con 0x7f00080b1730 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0008138900 con 0x7f00080b4d60 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00081377e0 con 0x7f000814be90 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:29.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.397+0000 7f0010417640 10 monclient(hunting): authenticate will time out at 1812.541911s 2024-03-25T14:27:29.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.402+0000 7f000e9b4640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f000814be90 0x7f0008144a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:48458/0 (socket says 172.21.15.63:48458) 2024-03-25T14:27:29.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.402+0000 7f000e9b4640 1 -- 172.21.15.63:0/1818553540 learned_addr learned my addr 172.21.15.63:0/1818553540 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:29.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.403+0000 7efffeffd640 10 client.?.objecter ms_handle_connect 0x7f000814be90 2024-03-25T14:27:29.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.403+0000 7efffeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:29.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.408+0000 7efffeffd640 10 client.?.objecter ms_handle_connect 0x7f00080b4d60 2024-03-25T14:27:29.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.408+0000 7efffeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:29.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.413+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019298586 0 0) 0x7f00081377e0 con 0x7f000814be90 2024-03-25T14:27:29.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.413+0000 7efffeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.413+0000 7efffeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.413+0000 7efffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.413+0000 7efffeffd640 10 monclient(hunting): my global_id is 4547 2024-03-25T14:27:29.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.413+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec004eb0 con 0x7f000814be90 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.420+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2398505186 0 0) 0x7effec004eb0 con 0x7f000814be90 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.420+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00081377e0 con 0x7f000814be90 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.420+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0000001c70 con 0x7f000814be90 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.420+0000 7efffeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.420+0000 7efffeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.420+0000 7efffeffd640 10 monclient(hunting): dump: 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.425 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.420+0000 7efffeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.422+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812463009 0 0) 0x7f0008138900 con 0x7f00080b4d60 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.422+0000 7efffeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.422+0000 7efffeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.422+0000 7efffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.422+0000 7efffeffd640 10 monclient(hunting): my global_id is 4573 2024-03-25T14:27:29.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.422+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec004eb0 con 0x7f00080b4d60 2024-03-25T14:27:29.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.433+0000 7efffeffd640 10 client.?.objecter ms_handle_connect 0x7f00080b1730 2024-03-25T14:27:29.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.433+0000 7efffeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:29.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.439+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969489389 0 0) 0x7f0008140890 con 0x7f00080b1730 2024-03-25T14:27:29.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.439+0000 7efffeffd640 10 monclient(hunting): _negotiate 2024-03-25T14:27:29.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.439+0000 7efffeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:29.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.439+0000 7efffeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:29.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.439+0000 7efffeffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T14:27:29.443 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.439+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0008138900 con 0x7f00080b1730 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659586573 0 0) 0x7effec004eb0 con 0x7f00080b4d60 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0008140890 con 0x7f00080b4d60 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7efff8001b00 con 0x7f00080b4d60 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 10 monclient(hunting): dump: 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.447 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 575829844 0 0) 0x7f00081377e0 con 0x7f000814be90 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 >> v1:172.21.15.153:6789/0 conn(0x7f00080b4d60 legacy=0x7f0008144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 >> v1:172.21.15.63:6789/0 conn(0x7f00080b1730 legacy=0x7f00081370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 10 monclient: _finish_hunting 0 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 monclient: found mon.c 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008137970 con 0x7f000814be90 2024-03-25T14:27:29.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 10 monclient: _finish_auth 0 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 10 monclient: _check_auth_tickets 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7efffeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 5 monclient: authenticate success, global_id 4547 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 10 monclient: _renew_subs 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0008137c20 con 0x7f000814be90 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 10 monclient: _renew_subs 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00081e39a0 con 0x7f000814be90 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.441+0000 7f0010417640 1 librados: init done 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.442+0000 7f0010417640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.442+0000 7f0010417640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.442+0000 7f0010417640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:29.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.442+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00080627b0 con 0x7f000814be90 2024-03-25T14:27:29.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.751+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0000004b60 con 0x7f000814be90 2024-03-25T14:27:29.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.751+0000 7efffeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:29.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.751+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0000004e70 con 0x7f000814be90 2024-03-25T14:27:29.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.751+0000 7efffeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:29.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.751+0000 7efffeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:29.756 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.751+0000 7efffeffd640 10 monclient: dump: 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:29.757 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.751+0000 7f000c9b0640 4 set_mon_vals no callback set 2024-03-25T14:27:29.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.752+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f000001ae40 con 0x7f000814be90 2024-03-25T14:27:29.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.753+0000 7f000e1b3640 10 monclient: get_auth_request con 0x7effec06e080 auth_method 0 2024-03-25T14:27:29.758 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.753+0000 7efffeffd640 10 client.4547.objecter ms_handle_connect 0x7effec06e080 2024-03-25T14:27:29.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f0000054a80 con 0x7f000814be90 2024-03-25T14:27:29.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 10 client.4547.objecter ms_dispatch 0x7f0008000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:29.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:29.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:29.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T14:27:29.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00000503c0 con 0x7f000814be90 2024-03-25T14:27:29.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:29.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:29.759+0000 7efffeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:30.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.017+0000 7f0010417640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:30.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.017+0000 7f0010417640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:30.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.017+0000 7f0010417640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:30.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.017+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00081357c0 con 0x7f000814be90 2024-03-25T14:27:30.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.051+0000 7efffeffd640 1 -- 172.21.15.63:0/1818553540 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f00000524c0 con 0x7f000814be90 2024-03-25T14:27:30.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.051+0000 7efffeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:30.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.051+0000 7efffeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:30.056 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:27:30.056 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:30.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.053+0000 7f0010417640 10 librados: watch_flush enter 2024-03-25T14:27:30.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 10 librados: watch_flush exit 2024-03-25T14:27:30.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T14:27:30.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 10 client.4547.objecter successfully canceled tick 2024-03-25T14:27:30.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 >> v1:172.21.15.153:6800/1688255552 conn(0x7effec06e080 legacy=0x7effec070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:30.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 10 monclient: shutdown 2024-03-25T14:27:30.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:30.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 >> v1:172.21.15.63:6790/0 conn(0x7f000814be90 legacy=0x7f0008144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:30.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 shutdown_connections 2024-03-25T14:27:30.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 >> 172.21.15.63:0/1818553540 conn(0x7f00080617a0 msgr2=0x7f000813c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:30.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 shutdown_connections 2024-03-25T14:27:30.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 1 -- 172.21.15.63:0/1818553540 wait complete. 2024-03-25T14:27:30.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:30.054+0000 7f0010417640 1 librados: shutdown 2024-03-25T14:27:30.070 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:36.876 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:36.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.989+0000 7ff4e52e6640 10 monclient: get_monmap_and_config 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.989+0000 7ff4e52e6640 10 monclient: build_initial_monmap 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.989+0000 7ff4e52e6640 10 monclient: monmap: 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:27:36.990612+0000 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:27:36.990612+0000 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:27:36.994 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:36.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.990+0000 7ff4e52e6640 1 Processor -- start 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.990+0000 7ff4e52e6640 1 -- start start 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.990+0000 7ff4e52e6640 10 monclient: init 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.990+0000 7ff4e52e6640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.990+0000 7ff4e52e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.990+0000 7ff4e52e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e00a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.990+0000 7ff4e52e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.991+0000 7ff4e52e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e01430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.991+0000 7ff4e52e6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e0142520 con 0x7ff4e013f710 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.991+0000 7ff4e52e6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e00b28a0 con 0x7ff4e01430c0 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.991+0000 7ff4e52e6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e00b2a30 con 0x7ff4e00a98f0 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.991+0000 7ff4e52e6640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:36.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:36.991+0000 7ff4e52e6640 10 monclient(hunting): authenticate will time out at 1820.135696s 2024-03-25T14:27:37.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.001+0000 7ff4df7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff4e01430c0 0x7ff4e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:40852/0 (socket says 172.21.15.63:40852) 2024-03-25T14:27:37.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.001+0000 7ff4df7fe640 1 -- 172.21.15.63:0/702305245 learned_addr learned my addr 172.21.15.63:0/702305245 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:37.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.012+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307747964 0 0) 0x7ff4e00b28a0 con 0x7ff4e01430c0 2024-03-25T14:27:37.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.012+0000 7ff4ddffb640 10 monclient(hunting): _negotiate 2024-03-25T14:27:37.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.012+0000 7ff4ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:37.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.012+0000 7ff4ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:37.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.012+0000 7ff4ddffb640 10 monclient(hunting): my global_id is 4576 2024-03-25T14:27:37.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.012+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c8003520 con 0x7ff4e01430c0 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285142008 0 0) 0x7ff4c8003520 con 0x7ff4e01430c0 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 >> v1:172.21.15.63:6790/0 conn(0x7ff4e00a98f0 legacy=0x7ff4e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 >> v1:172.21.15.63:6789/0 conn(0x7ff4e013f710 legacy=0x7ff4e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 10 monclient: _finish_hunting 0 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 1 monclient: found mon.noname-c 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:27:37.025 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e00b44c0 con 0x7ff4e01430c0 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 10 monclient: _finish_auth 0 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 10 monclient: _check_auth_tickets 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff4d8001a10 con 0x7ff4e01430c0 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4ddffb640 10 monclient: dump: 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.026 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4e52e6640 5 monclient: authenticate success, global_id 4576 2024-03-25T14:27:37.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.020+0000 7ff4e52e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4d8001be0 con 0x7ff4e01430c0 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4ddffb640 1 -- 172.21.15.63:0/702305245 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff4d8002030 con 0x7ff4e01430c0 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4ddffb640 10 monclient: dump: 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:37.031 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4e52e6640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:37.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.026+0000 7ff4e52e6640 4 set_mon_vals no callback set 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.027+0000 7ff4e52e6640 10 monclient: shutdown 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.027+0000 7ff4e52e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.027+0000 7ff4e52e6640 1 -- 172.21.15.63:0/702305245 >> v1:172.21.15.153:6789/0 conn(0x7ff4e01430c0 legacy=0x7ff4e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.027+0000 7ff4e52e6640 1 -- 172.21.15.63:0/702305245 shutdown_connections 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.027+0000 7ff4e52e6640 1 -- 172.21.15.63:0/702305245 >> 172.21.15.63:0/702305245 conn(0x7ff4e00b7ea0 msgr2=0x7ff4e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.027+0000 7ff4e52e6640 1 -- 172.21.15.63:0/702305245 shutdown_connections 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.027+0000 7ff4e52e6640 1 -- 172.21.15.63:0/702305245 wait complete. 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 10 monclient: build_initial_monmap 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 10 monclient: monmap: 2024-03-25T14:27:37.035 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 1 librados: starting msgr at 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 1 librados: starting objecter 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 1 Processor -- start 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 1 -- start start 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 1 librados: setting wanted keys 2024-03-25T14:27:37.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 1 librados: calling monclient init 2024-03-25T14:27:37.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.028+0000 7ff4e52e6640 10 monclient: init 2024-03-25T14:27:37.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:37.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:27:37.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e00a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:37.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e013f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:37.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e01430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:37.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e00ad4c0 con 0x7ff4e013f710 2024-03-25T14:27:37.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e00b16f0 con 0x7ff4e00a98f0 2024-03-25T14:27:37.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e01dacd0 con 0x7ff4e01430c0 2024-03-25T14:27:37.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:37.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.029+0000 7ff4e52e6640 10 monclient(hunting): authenticate will time out at 1820.174236s 2024-03-25T14:27:37.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.031+0000 7ff4de7fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff4e013f710 0x7ff4e00b0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59782/0 (socket says 172.21.15.63:59782) 2024-03-25T14:27:37.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.031+0000 7ff4de7fc640 1 -- 172.21.15.63:0/2372148157 learned_addr learned my addr 172.21.15.63:0/2372148157 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:37.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.032+0000 7ff4c77fe640 10 client.?.objecter ms_handle_connect 0x7ff4e013f710 2024-03-25T14:27:37.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.032+0000 7ff4c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:37.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.038+0000 7ff4c77fe640 10 client.?.objecter ms_handle_connect 0x7ff4e01430c0 2024-03-25T14:27:37.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.038+0000 7ff4c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:37.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 10 client.?.objecter ms_handle_connect 0x7ff4e00a98f0 2024-03-25T14:27:37.045 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929970912 0 0) 0x7ff4d40038c0 con 0x7ff4e013f710 2024-03-25T14:27:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 10 monclient(hunting): my global_id is 4572 2024-03-25T14:27:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.041+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4cc004eb0 con 0x7ff4e013f710 2024-03-25T14:27:37.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.042+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919682200 0 0) 0x7ff4e01dacd0 con 0x7ff4e01430c0 2024-03-25T14:27:37.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.042+0000 7ff4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:37.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.042+0000 7ff4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:37.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.042+0000 7ff4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:37.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.042+0000 7ff4c77fe640 10 monclient(hunting): my global_id is 4553 2024-03-25T14:27:37.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.042+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d40038c0 con 0x7ff4e01430c0 2024-03-25T14:27:37.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.047+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181714280 0 0) 0x7ff4e00b16f0 con 0x7ff4e00a98f0 2024-03-25T14:27:37.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.047+0000 7ff4c77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:37.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.047+0000 7ff4c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:37.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.047+0000 7ff4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:37.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.047+0000 7ff4c77fe640 10 monclient(hunting): my global_id is 4579 2024-03-25T14:27:37.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.047+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e01dacd0 con 0x7ff4e00a98f0 2024-03-25T14:27:37.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.053+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3221648433 0 0) 0x7ff4d40038c0 con 0x7ff4e01430c0 2024-03-25T14:27:37.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.053+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4e00b16f0 con 0x7ff4e01430c0 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.053+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff4d8001ef0 con 0x7ff4e01430c0 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.053+0000 7ff4c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.053+0000 7ff4c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.053+0000 7ff4c77fe640 10 monclient(hunting): dump: 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:37.058 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:37.059 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:37.059 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:37.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.053+0000 7ff4c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:37.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.058+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2334755085 0 0) 0x7ff4e00b16f0 con 0x7ff4e01430c0 2024-03-25T14:27:37.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.058+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 >> v1:172.21.15.153:6789/0 conn(0x7ff4e00a98f0 legacy=0x7ff4e01db1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.058+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 >> v1:172.21.15.63:6789/0 conn(0x7ff4e013f710 legacy=0x7ff4e00b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.058+0000 7ff4c77fe640 10 monclient: _finish_hunting 0 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.058+0000 7ff4c77fe640 1 monclient: found mon.c 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.058+0000 7ff4c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4c77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e014e780 con 0x7ff4e01430c0 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4c77fe640 10 monclient: _finish_auth 0 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4c77fe640 10 monclient: _check_auth_tickets 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 5 monclient: authenticate success, global_id 4553 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 10 monclient: _renew_subs 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e01da8f0 con 0x7ff4e01430c0 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:37.064 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 10 monclient: _renew_subs 2024-03-25T14:27:37.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:37.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e014dec0 con 0x7ff4e01430c0 2024-03-25T14:27:37.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.059+0000 7ff4e52e6640 1 librados: init done 2024-03-25T14:27:37.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.060+0000 7ff4e52e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:37.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.060+0000 7ff4e52e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:37.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.060+0000 7ff4e52e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:37.065 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.060+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e00b0a30 con 0x7ff4e01430c0 2024-03-25T14:27:37.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4d8002050 con 0x7ff4e01430c0 2024-03-25T14:27:37.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:37.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7ff4d8004fb0 con 0x7ff4e01430c0 2024-03-25T14:27:37.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4c77fe640 10 monclient: dump: 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:37.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4dd7fa640 4 set_mon_vals no callback set 2024-03-25T14:27:37.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.366+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7ff4d801af80 con 0x7ff4e01430c0 2024-03-25T14:27:37.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.367+0000 7ff4deffd640 10 monclient: get_auth_request con 0x7ff4cc06ddd0 auth_method 0 2024-03-25T14:27:37.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.367+0000 7ff4c77fe640 10 client.4553.objecter ms_handle_connect 0x7ff4cc06ddd0 2024-03-25T14:27:37.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.372+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7ff4d8054aa0 con 0x7ff4e01430c0 2024-03-25T14:27:37.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.373+0000 7ff4c77fe640 10 client.4553.objecter ms_dispatch 0x7ff4e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:37.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.373+0000 7ff4c77fe640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:37.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.373+0000 7ff4c77fe640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:37.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.373+0000 7ff4c77fe640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T14:27:37.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.373+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4d80505b0 con 0x7ff4e01430c0 2024-03-25T14:27:37.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.373+0000 7ff4c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:37.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.373+0000 7ff4c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:37.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.598+0000 7ff4e52e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:37.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.598+0000 7ff4e52e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:37.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.598+0000 7ff4e52e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:37.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.598+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4e00627b0 con 0x7ff4e01430c0 2024-03-25T14:27:37.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.663+0000 7ff4c77fe640 1 -- 172.21.15.63:0/2372148157 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7ff4d80524e0 con 0x7ff4e01430c0 2024-03-25T14:27:37.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.663+0000 7ff4c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:37.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.663+0000 7ff4c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:37.668 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:27:37.668 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.665+0000 7ff4e52e6640 10 librados: watch_flush enter 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.665+0000 7ff4e52e6640 10 librados: watch_flush exit 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.665+0000 7ff4e52e6640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.665+0000 7ff4e52e6640 10 client.4553.objecter successfully canceled tick 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 >> v1:172.21.15.153:6800/1688255552 conn(0x7ff4cc06ddd0 legacy=0x7ff4cc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 10 monclient: shutdown 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 >> v1:172.21.15.63:6790/0 conn(0x7ff4e01430c0 legacy=0x7ff4e01e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 shutdown_connections 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 >> 172.21.15.63:0/2372148157 conn(0x7ff4e00b7ea0 msgr2=0x7ff4e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 shutdown_connections 2024-03-25T14:27:37.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 1 -- 172.21.15.63:0/2372148157 wait complete. 2024-03-25T14:27:37.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:37.666+0000 7ff4e52e6640 1 librados: shutdown 2024-03-25T14:27:37.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:44.489 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.598+0000 7fc591b25640 10 monclient: get_monmap_and_config 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.598+0000 7fc591b25640 10 monclient: build_initial_monmap 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.598+0000 7fc591b25640 10 monclient: monmap: 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:27:44.599745+0000 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:27:44.599745+0000 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:44.603 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:44.604 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:44.604 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:27:44.604 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:27:44.604 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:44.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.599+0000 7fc591b25640 1 Processor -- start 2024-03-25T14:27:44.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.599+0000 7fc591b25640 1 -- start start 2024-03-25T14:27:44.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.599+0000 7fc591b25640 10 monclient: init 2024-03-25T14:27:44.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:44.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:27:44.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 10 monclient(hunting): picked mon.noname-a con 0x7fc58c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:44.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 10 monclient(hunting): picked mon.noname-c con 0x7fc58c13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:44.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:44.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c142520 con 0x7fc58c0a98f0 2024-03-25T14:27:44.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c0b28a0 con 0x7fc58c13f710 2024-03-25T14:27:44.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c0b2a30 con 0x7fc58c1430c0 2024-03-25T14:27:44.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:44.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.600+0000 7fc591b25640 10 monclient(hunting): authenticate will time out at 1827.744914s 2024-03-25T14:27:44.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.607+0000 7fc58b7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc58c0a98f0 0x7fc58c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:42966/0 (socket says 172.21.15.63:42966) 2024-03-25T14:27:44.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.607+0000 7fc58b7fe640 1 -- 172.21.15.63:0/572248663 learned_addr learned my addr 172.21.15.63:0/572248663 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:44.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.635+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918548849 0 0) 0x7fc58c0b28a0 con 0x7fc58c13f710 2024-03-25T14:27:44.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.635+0000 7fc58a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.635+0000 7fc58a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.635+0000 7fc58a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.635+0000 7fc58a7fc640 10 monclient(hunting): my global_id is 4582 2024-03-25T14:27:44.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.635+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc574003520 con 0x7fc58c13f710 2024-03-25T14:27:44.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.636+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549437803 0 0) 0x7fc58c142520 con 0x7fc58c0a98f0 2024-03-25T14:27:44.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.636+0000 7fc58a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.636+0000 7fc58a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.636+0000 7fc58a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.636+0000 7fc58a7fc640 10 monclient(hunting): my global_id is 4575 2024-03-25T14:27:44.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.636+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58c0b28a0 con 0x7fc58c0a98f0 2024-03-25T14:27:44.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.639+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028685852 0 0) 0x7fc58c0b2a30 con 0x7fc58c1430c0 2024-03-25T14:27:44.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.639+0000 7fc58a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.639+0000 7fc58a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.639+0000 7fc58a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.639+0000 7fc58a7fc640 10 monclient(hunting): my global_id is 4556 2024-03-25T14:27:44.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.639+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58c142520 con 0x7fc58c1430c0 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848814638 0 0) 0x7fc574003520 con 0x7fc58c13f710 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 >> v1:172.21.15.63:6790/0 conn(0x7fc58c1430c0 legacy=0x7fc58c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 >> v1:172.21.15.63:6789/0 conn(0x7fc58c0a98f0 legacy=0x7fc58c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 1 monclient: found mon.noname-c 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c0b44c0 con 0x7fc58c13f710 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:27:44.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc580001690 con 0x7fc58c13f710 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc58a7fc640 10 monclient: dump: 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:44.657 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:44.658 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:44.658 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:44.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc591b25640 5 monclient: authenticate success, global_id 4582 2024-03-25T14:27:44.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.651+0000 7fc591b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5800017f0 con 0x7fc58c13f710 2024-03-25T14:27:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc58a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc58a7fc640 1 -- 172.21.15.63:0/572248663 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc580001c00 con 0x7fc58c13f710 2024-03-25T14:27:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc591b25640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc591b25640 4 set_mon_vals no callback set 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc58a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc58a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.680+0000 7fc58a7fc640 10 monclient: dump: 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:44.686 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:44.687 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:44.687 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:44.687 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:44.687 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:44.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.681+0000 7fc591b25640 10 monclient: shutdown 2024-03-25T14:27:44.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.681+0000 7fc591b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:44.689 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.681+0000 7fc591b25640 1 -- 172.21.15.63:0/572248663 >> v1:172.21.15.153:6789/0 conn(0x7fc58c13f710 legacy=0x7fc58c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.681+0000 7fc591b25640 1 -- 172.21.15.63:0/572248663 shutdown_connections 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.681+0000 7fc591b25640 1 -- 172.21.15.63:0/572248663 >> 172.21.15.63:0/572248663 conn(0x7fc58c0b7ea0 msgr2=0x7fc58c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.681+0000 7fc591b25640 1 -- 172.21.15.63:0/572248663 shutdown_connections 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.681+0000 7fc591b25640 1 -- 172.21.15.63:0/572248663 wait complete. 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 10 monclient: build_initial_monmap 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 10 monclient: monmap: 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:27:44.690 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 1 librados: starting msgr at 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 1 librados: starting objecter 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 1 Processor -- start 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 1 -- start start 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 1 librados: setting wanted keys 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 1 librados: calling monclient init 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.682+0000 7fc591b25640 10 monclient: init 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:27:44.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 10 monclient(hunting): picked mon.noname-a con 0x7fc58c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:44.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 10 monclient(hunting): picked mon.noname-b con 0x7fc58c13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:44.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 10 monclient(hunting): picked mon.noname-c con 0x7fc58c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:44.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c138c10 con 0x7fc58c0a98f0 2024-03-25T14:27:44.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c138e60 con 0x7fc58c13f710 2024-03-25T14:27:44.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c139140 con 0x7fc58c1430c0 2024-03-25T14:27:44.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:44.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.683+0000 7fc591b25640 10 monclient(hunting): authenticate will time out at 1827.828241s 2024-03-25T14:27:44.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.690+0000 7fc58bfff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fc58c1430c0 0x7fc58c1e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:46142/0 (socket says 172.21.15.63:46142) 2024-03-25T14:27:44.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.690+0000 7fc58bfff640 1 -- 172.21.15.63:0/153466012 learned_addr learned my addr 172.21.15.63:0/153466012 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:44.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.691+0000 7fc573fff640 10 client.?.objecter ms_handle_connect 0x7fc58c1430c0 2024-03-25T14:27:44.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.691+0000 7fc573fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:44.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.696+0000 7fc573fff640 10 client.?.objecter ms_handle_connect 0x7fc58c0a98f0 2024-03-25T14:27:44.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.696+0000 7fc573fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.699+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173490285 0 0) 0x7fc58c139140 con 0x7fc58c1430c0 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.699+0000 7fc573fff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.699+0000 7fc573fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.699+0000 7fc573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.699+0000 7fc573fff640 10 monclient(hunting): my global_id is 4559 2024-03-25T14:27:44.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.700+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc578004eb0 con 0x7fc58c1430c0 2024-03-25T14:27:44.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.710+0000 7fc573fff640 10 client.?.objecter ms_handle_connect 0x7fc58c13f710 2024-03-25T14:27:44.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.710+0000 7fc573fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:44.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.713+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694210851 0 0) 0x7fc578004eb0 con 0x7fc58c1430c0 2024-03-25T14:27:44.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.713+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc58c139140 con 0x7fc58c1430c0 2024-03-25T14:27:44.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.713+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc584001c70 con 0x7fc58c1430c0 2024-03-25T14:27:44.718 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.713+0000 7fc573fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.713+0000 7fc573fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.713+0000 7fc573fff640 10 monclient(hunting): dump: 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:44.719 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.713+0000 7fc573fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:44.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.720+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412273304 0 0) 0x7fc58c138c10 con 0x7fc58c0a98f0 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.720+0000 7fc573fff640 10 monclient(hunting): _negotiate 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.720+0000 7fc573fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.720+0000 7fc573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.720+0000 7fc573fff640 10 monclient(hunting): my global_id is 4578 2024-03-25T14:27:44.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.720+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc578004eb0 con 0x7fc58c0a98f0 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 64736311 0 0) 0x7fc58c139140 con 0x7fc58c1430c0 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 >> v1:172.21.15.153:6789/0 conn(0x7fc58c13f710 legacy=0x7fc58c132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 >> v1:172.21.15.63:6789/0 conn(0x7fc58c0a98f0 legacy=0x7fc58c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 10 monclient: _finish_hunting 0 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 1 monclient: found mon.c 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c139920 con 0x7fc58c1430c0 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 10 monclient: _finish_auth 0 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 10 monclient: _check_auth_tickets 2024-03-25T14:27:44.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 5 monclient: authenticate success, global_id 4559 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 10 monclient: _renew_subs 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c1382e0 con 0x7fc58c1430c0 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 10 monclient: _renew_subs 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c0ab8b0 con 0x7fc58c1430c0 2024-03-25T14:27:44.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.727+0000 7fc591b25640 1 librados: init done 2024-03-25T14:27:44.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.728+0000 7fc591b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:44.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.728+0000 7fc591b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:44.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.728+0000 7fc591b25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:44.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:44.728+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c137a20 con 0x7fc58c1430c0 2024-03-25T14:27:45.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.047+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc584003990 con 0x7fc58c1430c0 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.047+0000 7fc573fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.047+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc584003de0 con 0x7fc58c1430c0 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.047+0000 7fc573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.047+0000 7fc573fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.047+0000 7fc573fff640 10 monclient: dump: 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:45.052 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:45.053 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:45.053 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:45.053 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:45.053 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:45.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.047+0000 7fc589ffb640 4 set_mon_vals no callback set 2024-03-25T14:27:45.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.048+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fc584004000 con 0x7fc58c1430c0 2024-03-25T14:27:45.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.049+0000 7fc58b7fe640 10 monclient: get_auth_request con 0x7fc57806dd90 auth_method 0 2024-03-25T14:27:45.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.049+0000 7fc573fff640 10 client.4559.objecter ms_handle_connect 0x7fc57806dd90 2024-03-25T14:27:45.059 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fc584054ab0 con 0x7fc58c1430c0 2024-03-25T14:27:45.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 10 client.4559.objecter ms_dispatch 0x7fc58c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:45.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:45.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:45.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T14:27:45.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5840505c0 con 0x7fc58c1430c0 2024-03-25T14:27:45.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:45.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.055+0000 7fc573fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:45.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.305+0000 7fc591b25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:45.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.305+0000 7fc591b25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:45.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.305+0000 7fc591b25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:27:45.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.305+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc58c13d010 con 0x7fc58c1430c0 2024-03-25T14:27:45.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.413+0000 7fc573fff640 1 -- 172.21.15.63:0/153466012 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fc5840524f0 con 0x7fc58c1430c0 2024-03-25T14:27:45.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.413+0000 7fc573fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:45.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.413+0000 7fc573fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:45.418 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:27:45.418 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:45.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 10 librados: watch_flush enter 2024-03-25T14:27:45.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 10 librados: watch_flush exit 2024-03-25T14:27:45.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 10 client.4559.objecter successfully canceled tick 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 >> v1:172.21.15.153:6800/1688255552 conn(0x7fc57806dd90 legacy=0x7fc578070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 10 monclient: shutdown 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 >> v1:172.21.15.63:6790/0 conn(0x7fc58c1430c0 legacy=0x7fc58c1e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 shutdown_connections 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.415+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 >> 172.21.15.63:0/153466012 conn(0x7fc58c0b7ea0 msgr2=0x7fc58c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.416+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 shutdown_connections 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.416+0000 7fc591b25640 1 -- 172.21.15.63:0/153466012 wait complete. 2024-03-25T14:27:45.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:45.416+0000 7fc591b25640 1 librados: shutdown 2024-03-25T14:27:45.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:47.411 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:25.923 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.417 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:25.929 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.445 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:25.958 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.462 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:25.974 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.475 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:25.934 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.478 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:25.990 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.483 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:25.943 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.508 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:25.968 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:47.510 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:26.022 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:27:52.237 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:27:52.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.342+0000 7fc4cac0e640 10 monclient: get_monmap_and_config 2024-03-25T14:27:52.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.343+0000 7fc4cac0e640 10 monclient: build_initial_monmap 2024-03-25T14:27:52.347 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.343+0000 7fc4cac0e640 10 monclient: monmap: 2024-03-25T14:27:52.347 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:52.347 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:27:52.344132+0000 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:27:52.344132+0000 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:27:52.348 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:52.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.344+0000 7fc4cac0e640 1 Processor -- start 2024-03-25T14:27:52.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.344+0000 7fc4cac0e640 1 -- start start 2024-03-25T14:27:52.349 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.344+0000 7fc4cac0e640 10 monclient: init 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c40b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c414be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40af940 con 0x7fc4c40b4d60 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40b0a70 con 0x7fc4c414be90 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40b0d50 con 0x7fc4c40b1730 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:52.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.345+0000 7fc4cac0e640 10 monclient(hunting): authenticate will time out at 1835.489910s 2024-03-25T14:27:52.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.355+0000 7fc4c89aa640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc4c40b4d60 0x7fc4c413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54918/0 (socket says 172.21.15.63:54918) 2024-03-25T14:27:52.359 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.355+0000 7fc4c89aa640 1 -- 172.21.15.63:0/2740665056 learned_addr learned my addr 172.21.15.63:0/2740665056 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:52.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.367+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298285501 0 0) 0x7fc4c40b0a70 con 0x7fc4c414be90 2024-03-25T14:27:52.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.367+0000 7fc4c37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.367+0000 7fc4c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.367+0000 7fc4c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.367+0000 7fc4c37fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T14:27:52.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.367+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ac003520 con 0x7fc4c414be90 2024-03-25T14:27:52.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.379+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773935308 0 0) 0x7fc4c40b0d50 con 0x7fc4c40b1730 2024-03-25T14:27:52.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.379+0000 7fc4c37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.379+0000 7fc4c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.379+0000 7fc4c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.379+0000 7fc4c37fe640 10 monclient(hunting): my global_id is 4562 2024-03-25T14:27:52.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.379+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c40b0a70 con 0x7fc4c40b1730 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537881791 0 0) 0x7fc4ac003520 con 0x7fc4c414be90 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 >> v1:172.21.15.63:6790/0 conn(0x7fc4c40b1730 legacy=0x7fc4c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 >> v1:172.21.15.63:6789/0 conn(0x7fc4c40b4d60 legacy=0x7fc4c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 10 monclient: _finish_hunting 0 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 1 monclient: found mon.noname-c 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c40b1030 con 0x7fc4c414be90 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 10 monclient: _finish_auth 0 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 10 monclient: _check_auth_tickets 2024-03-25T14:27:52.386 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc4bc001a10 con 0x7fc4c414be90 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4c37fe640 10 monclient: dump: 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:52.387 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:52.388 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:52.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4cac0e640 5 monclient: authenticate success, global_id 4588 2024-03-25T14:27:52.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.380+0000 7fc4cac0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.385+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4bc001b70 con 0x7fc4c414be90 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.385+0000 7fc4c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.385+0000 7fc4cac0e640 10 monclient: get_monmap_and_config success 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.385+0000 7fc4cac0e640 4 set_mon_vals no callback set 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4c37fe640 1 -- 172.21.15.63:0/2740665056 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc4bc001f80 con 0x7fc4c414be90 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4c37fe640 10 monclient: dump: 2024-03-25T14:27:52.390 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:52.391 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:52.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4cac0e640 10 monclient: shutdown 2024-03-25T14:27:52.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4cac0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4cac0e640 1 -- 172.21.15.63:0/2740665056 >> v1:172.21.15.153:6789/0 conn(0x7fc4c414be90 legacy=0x7fc4c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4cac0e640 1 -- 172.21.15.63:0/2740665056 shutdown_connections 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4cac0e640 1 -- 172.21.15.63:0/2740665056 >> 172.21.15.63:0/2740665056 conn(0x7fc4c40617a0 msgr2=0x7fc4c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4cac0e640 1 -- 172.21.15.63:0/2740665056 shutdown_connections 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.386+0000 7fc4cac0e640 1 -- 172.21.15.63:0/2740665056 wait complete. 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 10 monclient: build_initial_monmap 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 10 monclient: monmap: 2024-03-25T14:27:52.394 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:27:52.398 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:27:52.398 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:27:52.398 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:27:52.398 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:27:52.398 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:27:52.398 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 1 librados: starting msgr at 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 1 librados: starting objecter 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 1 Processor -- start 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 1 -- start start 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 1 librados: setting wanted keys 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 1 librados: calling monclient init 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.387+0000 7fc4cac0e640 10 monclient: init 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 10 monclient: _reopen_session rank -1 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c40b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:27:52.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c40b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:27:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c414be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:27:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c4138a00 con 0x7fc4c40b1730 2024-03-25T14:27:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c4135b00 con 0x7fc4c40b4d60 2024-03-25T14:27:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c4135c90 con 0x7fc4c414be90 2024-03-25T14:27:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 10 monclient(hunting): _renew_subs 2024-03-25T14:27:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.388+0000 7fc4cac0e640 10 monclient(hunting): authenticate will time out at 1835.533353s 2024-03-25T14:27:52.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.394+0000 7fc4c3fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc4c40b4d60 0x7fc4c41382f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:56854/0 (socket says 172.21.15.63:56854) 2024-03-25T14:27:52.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.395+0000 7fc4c3fff640 1 -- 172.21.15.63:0/860567171 learned_addr learned my addr 172.21.15.63:0/860567171 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:27:52.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.395+0000 7fc4c17fa640 10 client.?.objecter ms_handle_connect 0x7fc4c40b4d60 2024-03-25T14:27:52.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.395+0000 7fc4c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:52.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.397+0000 7fc4c3fff640 1 -- 172.21.15.63:0/860567171 >> v1:172.21.15.153:6789/0 conn(0x7fc4c40b4d60 legacy=0x7fc4c41382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T14:27:52.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.397+0000 7fc4c3fff640 1 -- 172.21.15.63:0/860567171 >> v1:172.21.15.153:6789/0 conn(0x7fc4c40b4d60 legacy=0x7fc4c41382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:27:52.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.397+0000 7fc4c3fff640 1 --1- 172.21.15.63:0/860567171 >> v1:172.21.15.153:6789/0 conn(0x7fc4c40b4d60 0x7fc4c41382f0 :-1 s=OPENED pgs=57337 cs=1 l=1).handle_message read tag failed 2024-03-25T14:27:52.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.397+0000 7fc4c3fff640 1 --1- 172.21.15.63:0/860567171 >> v1:172.21.15.153:6789/0 conn(0x7fc4c40b4d60 0x7fc4c41382f0 :-1 s=OPENED pgs=57337 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:27:52.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.397+0000 7fc4c17fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6789/0 2024-03-25T14:27:52.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.403+0000 7fc4c17fa640 10 client.?.objecter ms_handle_connect 0x7fc4c414be90 2024-03-25T14:27:52.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.403+0000 7fc4c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:52.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.410+0000 7fc4c17fa640 10 client.?.objecter ms_handle_connect 0x7fc4c40b1730 2024-03-25T14:27:52.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.410+0000 7fc4c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:27:52.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.468+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323205461 0 0) 0x7fc4c40af940 con 0x7fc4c40b1730 2024-03-25T14:27:52.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.468+0000 7fc4c17fa640 10 monclient(hunting): _negotiate 2024-03-25T14:27:52.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.468+0000 7fc4c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:27:52.473 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.468+0000 7fc4c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:27:52.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.468+0000 7fc4c17fa640 10 monclient(hunting): my global_id is 4584 2024-03-25T14:27:52.474 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.469+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b0004eb0 con 0x7fc4c40b1730 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.486+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 989136324 0 0) 0x7fc4b0004eb0 con 0x7fc4c40b1730 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.486+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c40af940 con 0x7fc4c40b1730 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.486+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc4b4001700 con 0x7fc4c40b1730 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.486+0000 7fc4c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.486+0000 7fc4c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.486+0000 7fc4c17fa640 10 monclient(hunting): dump: 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.491 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.492 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:52.492 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:52.492 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:52.492 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:52.492 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:52.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.486+0000 7fc4c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:27:52.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4145794057 0 0) 0x7fc4c40af940 con 0x7fc4c40b1730 2024-03-25T14:27:52.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 >> v1:172.21.15.63:6790/0 conn(0x7fc4c414be90 legacy=0x7fc4c41e2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:52.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 >> v1:172.21.15.153:6789/0 conn(0x7fc4c40b4d60 legacy=0x7fc4c41382f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:27:52.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 10 monclient: _finish_hunting 0 2024-03-25T14:27:52.510 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 1 monclient: found mon.a 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4136410 con 0x7fc4c40b1730 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 10 monclient: _finish_auth 0 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 10 monclient: _check_auth_tickets 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 5 monclient: authenticate success, global_id 4584 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 10 monclient: _renew_subs 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c4145160 con 0x7fc4c40b1730 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:27:52.511 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 10 monclient: _renew_subs 2024-03-25T14:27:52.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:27:52.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c413e690 con 0x7fc4c40b1730 2024-03-25T14:27:52.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.505+0000 7fc4cac0e640 1 librados: init done 2024-03-25T14:27:52.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.506+0000 7fc4cac0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:52.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.506+0000 7fc4cac0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:52.512 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.506+0000 7fc4cac0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:27:52.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.506+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c40627b0 con 0x7fc4c40b1730 2024-03-25T14:27:52.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.821+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b4001b70 con 0x7fc4c40b1730 2024-03-25T14:27:52.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.821+0000 7fc4c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:27:52.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.821+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fc4b4004980 con 0x7fc4c40b1730 2024-03-25T14:27:52.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.821+0000 7fc4c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:27:52.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.821+0000 7fc4c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:27:52.826 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.821+0000 7fc4c17fa640 10 monclient: dump: 2024-03-25T14:27:52.826 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:27:52.827 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.821+0000 7fc4c2ffd640 4 set_mon_vals no callback set 2024-03-25T14:27:52.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.822+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fc4b4004ba0 con 0x7fc4c40b1730 2024-03-25T14:27:52.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.823+0000 7fc4c3fff640 10 monclient: get_auth_request con 0x7fc4b006dc70 auth_method 0 2024-03-25T14:27:52.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.823+0000 7fc4c17fa640 10 client.4584.objecter ms_handle_connect 0x7fc4b006dc70 2024-03-25T14:27:52.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fc4b40545f0 con 0x7fc4c40b1730 2024-03-25T14:27:52.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 10 client.4584.objecter ms_dispatch 0x7fc4c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:27:52.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:27:52.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:27:52.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-25T14:27:52.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4b4050100 con 0x7fc4c40b1730 2024-03-25T14:27:52.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:27:52.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:52.831+0000 7fc4c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:27:53.086 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.082+0000 7fc4cac0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:53.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.082+0000 7fc4cac0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:53.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.082+0000 7fc4cac0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:27:53.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.082+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4c4069f40 con 0x7fc4c40b1730 2024-03-25T14:27:53.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.119+0000 7fc4c17fa640 1 -- 172.21.15.63:0/860567171 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fc4b4052030 con 0x7fc4c40b1730 2024-03-25T14:27:53.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.119+0000 7fc4c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:27:53.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.119+0000 7fc4c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:27:53.124 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:27:53.125 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:27:53.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 10 librados: watch_flush enter 2024-03-25T14:27:53.126 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 10 librados: watch_flush exit 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 10 client.4584.objecter successfully canceled tick 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 >> v1:172.21.15.153:6800/1688255552 conn(0x7fc4b006dc70 legacy=0x7fc4b0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 10 monclient: shutdown 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 >> v1:172.21.15.63:6789/0 conn(0x7fc4c40b1730 legacy=0x7fc4c4137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 shutdown_connections 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.121+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 >> 172.21.15.63:0/860567171 conn(0x7fc4c40617a0 msgr2=0x7fc4c413c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.122+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 shutdown_connections 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.122+0000 7fc4cac0e640 1 -- 172.21.15.63:0/860567171 wait complete. 2024-03-25T14:27:53.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:27:53.122+0000 7fc4cac0e640 1 librados: shutdown 2024-03-25T14:27:53.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:27:59.947 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:00.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.056+0000 7fcf8e53d640 10 monclient: get_monmap_and_config 2024-03-25T14:28:00.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.056+0000 7fcf8e53d640 10 monclient: build_initial_monmap 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.056+0000 7fcf8e53d640 10 monclient: monmap: 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:00.057308+0000 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:00.057308+0000 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:00.061 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:00.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.056+0000 7fcf8e53d640 1 Processor -- start 2024-03-25T14:28:00.062 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 1 -- start start 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient: init 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient(hunting): picked mon.noname-b con 0x7fcf880b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient(hunting): picked mon.noname-a con 0x7fcf880b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient(hunting): picked mon.noname-c con 0x7fcf8814be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880af940 con 0x7fcf880b1730 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880b0a70 con 0x7fcf8814be90 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880b0d50 con 0x7fcf880b4d60 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:00.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.057+0000 7fcf8e53d640 10 monclient(hunting): authenticate will time out at 1843.202248s 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.072+0000 7fcf87fff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fcf880b4d60 0x7fcf8813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:58902/0 (socket says 172.21.15.63:58902) 2024-03-25T14:28:00.076 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.072+0000 7fcf87fff640 1 -- 172.21.15.63:0/591466134 learned_addr learned my addr 172.21.15.63:0/591466134 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:00.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.084+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238817520 0 0) 0x7fcf880b0a70 con 0x7fcf8814be90 2024-03-25T14:28:00.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.084+0000 7fcf86ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.084+0000 7fcf86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.084+0000 7fcf86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.084+0000 7fcf86ffd640 10 monclient(hunting): my global_id is 4591 2024-03-25T14:28:00.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.084+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf70003520 con 0x7fcf8814be90 2024-03-25T14:28:00.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.089+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620301291 0 0) 0x7fcf880b0d50 con 0x7fcf880b4d60 2024-03-25T14:28:00.093 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.089+0000 7fcf86ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.089+0000 7fcf86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.089+0000 7fcf86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.089+0000 7fcf86ffd640 10 monclient(hunting): my global_id is 4568 2024-03-25T14:28:00.094 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.089+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf880b0a70 con 0x7fcf880b4d60 2024-03-25T14:28:00.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.094+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652785909 0 0) 0x7fcf880af940 con 0x7fcf880b1730 2024-03-25T14:28:00.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.094+0000 7fcf86ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.094+0000 7fcf86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.094+0000 7fcf86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.094+0000 7fcf86ffd640 10 monclient(hunting): my global_id is 4587 2024-03-25T14:28:00.098 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.094+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf880b0d50 con 0x7fcf880b1730 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2148156438 0 0) 0x7fcf880b0a70 con 0x7fcf880b4d60 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 >> v1:172.21.15.153:6789/0 conn(0x7fcf8814be90 legacy=0x7fcf8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 >> v1:172.21.15.63:6789/0 conn(0x7fcf880b1730 legacy=0x7fcf880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 1 monclient: found mon.noname-b 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:28:00.113 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880b1030 con 0x7fcf880b4d60 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 10 monclient: _finish_auth 0 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcf78001670 con 0x7fcf880b4d60 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf86ffd640 10 monclient: dump: 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:00.114 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:00.115 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:00.115 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:00.115 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:00.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf8e53d640 5 monclient: authenticate success, global_id 4568 2024-03-25T14:28:00.115 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.108+0000 7fcf8e53d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf780017d0 con 0x7fcf880b4d60 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf86ffd640 1 -- 172.21.15.63:0/591466134 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcf78001be0 con 0x7fcf880b4d60 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf8e53d640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf8e53d640 4 set_mon_vals no callback set 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.131+0000 7fcf86ffd640 10 monclient: dump: 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.136 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.137 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.137 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:00.137 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:00.137 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:00.137 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:00.137 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:00.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.132+0000 7fcf8e53d640 10 monclient: shutdown 2024-03-25T14:28:00.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.132+0000 7fcf8e53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:00.140 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.132+0000 7fcf8e53d640 1 -- 172.21.15.63:0/591466134 >> v1:172.21.15.63:6790/0 conn(0x7fcf880b4d60 legacy=0x7fcf8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.132+0000 7fcf8e53d640 1 -- 172.21.15.63:0/591466134 shutdown_connections 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.132+0000 7fcf8e53d640 1 -- 172.21.15.63:0/591466134 >> 172.21.15.63:0/591466134 conn(0x7fcf880617a0 msgr2=0x7fcf88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.132+0000 7fcf8e53d640 1 -- 172.21.15.63:0/591466134 shutdown_connections 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.132+0000 7fcf8e53d640 1 -- 172.21.15.63:0/591466134 wait complete. 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.133+0000 7fcf8e53d640 10 monclient: build_initial_monmap 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.133+0000 7fcf8e53d640 10 monclient: monmap: 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:00.141 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.133+0000 7fcf8e53d640 1 librados: starting msgr at 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.133+0000 7fcf8e53d640 1 librados: starting objecter 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.134+0000 7fcf8e53d640 1 Processor -- start 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.134+0000 7fcf8e53d640 1 -- start start 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.134+0000 7fcf8e53d640 1 librados: setting wanted keys 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.134+0000 7fcf8e53d640 1 librados: calling monclient init 2024-03-25T14:28:00.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.134+0000 7fcf8e53d640 10 monclient: init 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 10 monclient(hunting): picked mon.noname-c con 0x7fcf880b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 10 monclient(hunting): picked mon.noname-a con 0x7fcf880b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 10 monclient(hunting): picked mon.noname-b con 0x7fcf8814be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf88137e40 con 0x7fcf880b4d60 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf88143900 con 0x7fcf8814be90 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf88143a90 con 0x7fcf880b1730 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:00.143 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.135+0000 7fcf8e53d640 10 monclient(hunting): authenticate will time out at 1843.279730s 2024-03-25T14:28:00.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.141+0000 7fcf8cada640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcf8814be90 0x7fcf88136480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:50322/0 (socket says 172.21.15.63:50322) 2024-03-25T14:28:00.145 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.141+0000 7fcf8cada640 1 -- 172.21.15.63:0/2132269532 learned_addr learned my addr 172.21.15.63:0/2132269532 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:00.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.145+0000 7fcf84ff9640 10 client.?.objecter ms_handle_connect 0x7fcf8814be90 2024-03-25T14:28:00.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.145+0000 7fcf84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:00.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.146+0000 7fcf84ff9640 10 client.?.objecter ms_handle_connect 0x7fcf880b1730 2024-03-25T14:28:00.150 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.146+0000 7fcf84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:00.156 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.152+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153196313 0 0) 0x7fcf88143900 con 0x7fcf8814be90 2024-03-25T14:28:00.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.152+0000 7fcf84ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.152+0000 7fcf84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.152+0000 7fcf84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.152+0000 7fcf84ff9640 10 monclient(hunting): my global_id is 4594 2024-03-25T14:28:00.157 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.152+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf74004eb0 con 0x7fcf8814be90 2024-03-25T14:28:00.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.160+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3911474424 0 0) 0x7fcf74004eb0 con 0x7fcf8814be90 2024-03-25T14:28:00.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.160+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf88143900 con 0x7fcf8814be90 2024-03-25T14:28:00.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.160+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcf80001c30 con 0x7fcf8814be90 2024-03-25T14:28:00.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.160+0000 7fcf84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.160+0000 7fcf84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:00.165 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.160+0000 7fcf84ff9640 10 monclient(hunting): dump: 2024-03-25T14:28:00.165 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:00.166 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.160+0000 7fcf84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:00.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.162+0000 7fcf84ff9640 10 client.?.objecter ms_handle_connect 0x7fcf880b4d60 2024-03-25T14:28:00.167 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.162+0000 7fcf84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:00.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.163+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706957257 0 0) 0x7fcf88143a90 con 0x7fcf880b1730 2024-03-25T14:28:00.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.163+0000 7fcf84ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.163+0000 7fcf84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.163+0000 7fcf84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:00.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.163+0000 7fcf84ff9640 10 monclient(hunting): my global_id is 4571 2024-03-25T14:28:00.168 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.163+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf74004eb0 con 0x7fcf880b1730 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2088558734 0 0) 0x7fcf88143900 con 0x7fcf8814be90 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.63:6790/0 conn(0x7fcf880b1730 legacy=0x7fcf881db310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.63:6789/0 conn(0x7fcf880b4d60 legacy=0x7fcf88135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 1 monclient: found mon.b 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf881dba20 con 0x7fcf8814be90 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 10 monclient: _finish_auth 0 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:28:00.173 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 5 monclient: authenticate success, global_id 4594 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 10 monclient: _renew_subs 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf881dbc70 con 0x7fcf8814be90 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 10 monclient: _renew_subs 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf881e34a0 con 0x7fcf8814be90 2024-03-25T14:28:00.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.168+0000 7fcf8e53d640 1 librados: init done 2024-03-25T14:28:00.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.169+0000 7fcf8e53d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:00.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.169+0000 7fcf8e53d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:00.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.169+0000 7fcf8e53d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:00.175 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.169+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf881401f0 con 0x7fcf8814be90 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.190+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf80003990 con 0x7fcf8814be90 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.190+0000 7fcf84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.190+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcf80003da0 con 0x7fcf8814be90 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.190+0000 7fcf84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.190+0000 7fcf84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.190+0000 7fcf84ff9640 10 monclient: dump: 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:00.195 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:00.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.190+0000 7fcf867fc640 4 set_mon_vals no callback set 2024-03-25T14:28:00.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.191+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fcf80003fc0 con 0x7fcf8814be90 2024-03-25T14:28:00.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.192+0000 7fcf87fff640 10 monclient: get_auth_request con 0x7fcf7406dd60 auth_method 0 2024-03-25T14:28:00.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.192+0000 7fcf84ff9640 10 client.4594.objecter ms_handle_connect 0x7fcf7406dd60 2024-03-25T14:28:00.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.196+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fcf800549e0 con 0x7fcf8814be90 2024-03-25T14:28:00.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.196+0000 7fcf84ff9640 10 client.4594.objecter ms_dispatch 0x7fcf88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:00.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.196+0000 7fcf84ff9640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:00.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.196+0000 7fcf84ff9640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:00.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.196+0000 7fcf84ff9640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T14:28:00.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf8cada640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.153:6789/0 conn(0x7fcf8814be90 legacy=0x7fcf88136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T14:28:00.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf8cada640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.153:6789/0 conn(0x7fcf8814be90 legacy=0x7fcf88136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:28:00.202 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf8cada640 1 --1- 172.21.15.63:0/2132269532 >> v1:172.21.15.153:6789/0 conn(0x7fcf8814be90 0x7fcf88136480 :-1 s=OPENED pgs=58400 cs=1 l=1).handle_message read tag failed 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf8cada640 1 --1- 172.21.15.63:0/2132269532 >> v1:172.21.15.153:6789/0 conn(0x7fcf8814be90 0x7fcf88136480 :-1 s=OPENED pgs=58400 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.153:6789/0 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.153:6789/0 conn(0x7fcf8814be90 legacy=0x7fcf88136480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 10 monclient(hunting): picked mon.c con 0x7fcf880b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 10 monclient(hunting): picked mon.a con 0x7fcf880b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:00.203 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 10 monclient(hunting): picked mon.b con 0x7fcf7407a0b0 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:00.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcf88143900 con 0x7fcf880b4d60 2024-03-25T14:28:00.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcf88143a90 con 0x7fcf7407a0b0 2024-03-25T14:28:00.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fcf74080290 con 0x7fcf880b1730 2024-03-25T14:28:00.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.197+0000 7fcf84ff9640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:00.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.208+0000 7fcf84ff9640 10 client.4594.objecter ms_handle_connect 0x7fcf880b4d60 2024-03-25T14:28:00.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.208+0000 7fcf84ff9640 10 client.4594.objecter resend_mon_ops 2024-03-25T14:28:00.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.215+0000 7fcf84ff9640 10 client.4594.objecter ms_handle_connect 0x7fcf880b1730 2024-03-25T14:28:00.220 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.215+0000 7fcf84ff9640 10 client.4594.objecter resend_mon_ops 2024-03-25T14:28:00.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.221+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740616726 0 0) 0x7fcf88143900 con 0x7fcf880b4d60 2024-03-25T14:28:00.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.221+0000 7fcf84ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.221+0000 7fcf84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.221+0000 7fcf84ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T14:28:00.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.221+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fcf74080420 con 0x7fcf880b4d60 2024-03-25T14:28:00.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.226+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773085340 0 0) 0x7fcf74080290 con 0x7fcf880b1730 2024-03-25T14:28:00.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.226+0000 7fcf84ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:28:00.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.226+0000 7fcf84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:00.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.226+0000 7fcf84ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T14:28:00.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.226+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fcf88143900 con 0x7fcf880b1730 2024-03-25T14:28:00.250 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3711022203 0 0) 0x7fcf88143900 con 0x7fcf880b1730 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.153:6789/0 conn(0x7fcf7407a0b0 legacy=0x7fcf7407c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.63:6789/0 conn(0x7fcf880b4d60 legacy=0x7fcf74078460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 1 monclient: found mon.c 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fcf740814f0 con 0x7fcf880b1730 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf88143a90 con 0x7fcf880b1730 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 10 monclient: _finish_auth 0 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:28:00.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.246+0000 7fcf84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:00.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.252+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fcf7c0015a0 con 0x7fcf880b1730 2024-03-25T14:28:00.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.252+0000 7fcf84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:00.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.252+0000 7fcf84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:00.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.252+0000 7fcf84ff9640 10 monclient: dump: 2024-03-25T14:28:00.256 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:00.256 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:00.257 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:00.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.569+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf7c001400 con 0x7fcf880b1730 2024-03-25T14:28:00.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.569+0000 7fcf84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:00.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.569+0000 7fcf85ffb640 4 set_mon_vals no callback set 2024-03-25T14:28:00.575 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.570+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fcf7c001840 con 0x7fcf880b1730 2024-03-25T14:28:00.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.575+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf7c052ba0 con 0x7fcf880b1730 2024-03-25T14:28:00.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.575+0000 7fcf84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:00.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.575+0000 7fcf84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:00.821 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.817+0000 7fcf8e53d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:00.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.817+0000 7fcf8e53d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:00.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.817+0000 7fcf8e53d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:28:00.822 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.817+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf881daff0 con 0x7fcf880b1730 2024-03-25T14:28:00.873 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.868+0000 7fcf84ff9640 1 -- 172.21.15.63:0/2132269532 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fcf7c052ba0 con 0x7fcf880b1730 2024-03-25T14:28:00.873 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.868+0000 7fcf84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:00.873 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.868+0000 7fcf84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:00.874 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:00.874 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:00.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 10 librados: watch_flush enter 2024-03-25T14:28:00.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 10 librados: watch_flush exit 2024-03-25T14:28:00.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T14:28:00.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 10 client.4594.objecter successfully canceled tick 2024-03-25T14:28:00.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.153:6800/1688255552 conn(0x7fcf7406dd60 legacy=0x7fcf74070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 10 monclient: shutdown 2024-03-25T14:28:00.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:00.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 >> v1:172.21.15.63:6790/0 conn(0x7fcf880b1730 legacy=0x7fcf74074a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:00.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 shutdown_connections 2024-03-25T14:28:00.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 >> 172.21.15.63:0/2132269532 conn(0x7fcf880617a0 msgr2=0x7fcf880b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:00.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 shutdown_connections 2024-03-25T14:28:00.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 1 -- 172.21.15.63:0/2132269532 wait complete. 2024-03-25T14:28:00.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:00.871+0000 7fcf8e53d640 1 librados: shutdown 2024-03-25T14:28:00.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:07.560 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:46.072 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.568 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:46.081 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.594 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:08:46.106 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.616 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:46.076 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.620 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:46.080 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.637 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:46.149 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.642 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:46.102 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.646 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:46.158 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.680 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:08:46.192 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:07.694 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.801+0000 7f68202c2640 10 monclient: get_monmap_and_config 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.801+0000 7f68202c2640 10 monclient: build_initial_monmap 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.801+0000 7f68202c2640 10 monclient: monmap: 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:07.801726+0000 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:07.801726+0000 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:07.805 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:07.806 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:07.806 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:07.806 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:07.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.802+0000 7f68202c2640 1 Processor -- start 2024-03-25T14:28:07.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.802+0000 7f68202c2640 1 -- start start 2024-03-25T14:28:07.807 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.802+0000 7f68202c2640 10 monclient: init 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 10 monclient(hunting): picked mon.noname-b con 0x7f68180b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 10 monclient(hunting): picked mon.noname-c con 0x7f68180b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 10 monclient(hunting): picked mon.noname-a con 0x7f681814be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180af940 con 0x7f681814be90 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180b0a70 con 0x7f68180b1730 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180b0d50 con 0x7f68180b4d60 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:07.808 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.803+0000 7f68202c2640 10 monclient(hunting): authenticate will time out at 1850.946689s 2024-03-25T14:28:07.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.825+0000 7f681e05e640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f68180b4d60 0x7f681813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:35394/0 (socket says 172.21.15.63:35394) 2024-03-25T14:28:07.829 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.825+0000 7f681e05e640 1 -- 172.21.15.63:0/1699703952 learned_addr learned my addr 172.21.15.63:0/1699703952 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:07.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.848+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250024904 0 0) 0x7f68180b0a70 con 0x7f68180b1730 2024-03-25T14:28:07.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.848+0000 7f681d05c640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.848+0000 7f681d05c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.848+0000 7f681d05c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.852 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.848+0000 7f681d05c640 10 monclient(hunting): my global_id is 4597 2024-03-25T14:28:07.853 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.848+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6804003520 con 0x7f68180b1730 2024-03-25T14:28:07.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.851+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399397177 0 0) 0x7f68180af940 con 0x7f681814be90 2024-03-25T14:28:07.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.851+0000 7f681d05c640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.851+0000 7f681d05c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.851+0000 7f681d05c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.854 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.851+0000 7f681d05c640 10 monclient(hunting): my global_id is 4593 2024-03-25T14:28:07.855 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.851+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68180b0a70 con 0x7f681814be90 2024-03-25T14:28:07.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.857+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464819491 0 0) 0x7f68180b0d50 con 0x7f68180b4d60 2024-03-25T14:28:07.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.857+0000 7f681d05c640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.857+0000 7f681d05c640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.857+0000 7f681d05c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.857+0000 7f681d05c640 10 monclient(hunting): my global_id is 4574 2024-03-25T14:28:07.861 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.857+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68180af940 con 0x7f68180b4d60 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 824419403 0 0) 0x7f6804003520 con 0x7f68180b1730 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 >> v1:172.21.15.63:6790/0 conn(0x7f68180b4d60 legacy=0x7f681813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 >> v1:172.21.15.63:6789/0 conn(0x7f681814be90 legacy=0x7f681814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 10 monclient: _finish_hunting 0 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 1 monclient: found mon.noname-c 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68180b1030 con 0x7f68180b1730 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 10 monclient: _finish_auth 0 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 10 monclient: _check_auth_tickets 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6808001690 con 0x7f68180b1730 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f681d05c640 10 monclient: dump: 2024-03-25T14:28:07.880 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f68202c2640 5 monclient: authenticate success, global_id 4597 2024-03-25T14:28:07.881 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.875+0000 7f68202c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:07.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68080017f0 con 0x7f68180b1730 2024-03-25T14:28:07.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f681d05c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:07.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f681d05c640 1 -- 172.21.15.63:0/1699703952 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6808001c00 con 0x7f68180b1730 2024-03-25T14:28:07.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f681d05c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f681d05c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f681d05c640 10 monclient: dump: 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:07.907 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:07.908 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 4 set_mon_vals no callback set 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 10 monclient: shutdown 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 1 -- 172.21.15.63:0/1699703952 >> v1:172.21.15.153:6789/0 conn(0x7f68180b1730 legacy=0x7f68180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 1 -- 172.21.15.63:0/1699703952 shutdown_connections 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 1 -- 172.21.15.63:0/1699703952 >> 172.21.15.63:0/1699703952 conn(0x7f68180617a0 msgr2=0x7f6818061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 1 -- 172.21.15.63:0/1699703952 shutdown_connections 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.903+0000 7f68202c2640 1 -- 172.21.15.63:0/1699703952 wait complete. 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.904+0000 7f68202c2640 10 monclient: build_initial_monmap 2024-03-25T14:28:07.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.904+0000 7f68202c2640 10 monclient: monmap: 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.904+0000 7f68202c2640 1 librados: starting msgr at 2024-03-25T14:28:07.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.904+0000 7f68202c2640 1 librados: starting objecter 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.904+0000 7f68202c2640 1 Processor -- start 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 1 -- start start 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 1 librados: setting wanted keys 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 1 librados: calling monclient init 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient: init 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient(hunting): picked mon.noname-b con 0x7f68180b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient(hunting): picked mon.noname-a con 0x7f68180b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient(hunting): picked mon.noname-c con 0x7f681814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68181d83d0 con 0x7f68180b4d60 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68181d77a0 con 0x7f68180b1730 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6818138270 con 0x7f681814be90 2024-03-25T14:28:07.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:07.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.905+0000 7f68202c2640 10 monclient(hunting): authenticate will time out at 1851.049198s 2024-03-25T14:28:07.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.925+0000 7f681d85d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f68180b4d60 0x7f68181405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:59964/0 (socket says 172.21.15.63:59964) 2024-03-25T14:28:07.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.925+0000 7f681d85d640 1 -- 172.21.15.63:0/3813760559 learned_addr learned my addr 172.21.15.63:0/3813760559 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:07.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.926+0000 7f680effd640 10 client.?.objecter ms_handle_connect 0x7f68180b1730 2024-03-25T14:28:07.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.926+0000 7f680effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:07.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.927+0000 7f680effd640 10 client.?.objecter ms_handle_connect 0x7f68180b4d60 2024-03-25T14:28:07.930 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.927+0000 7f680effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:07.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.937+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062914557 0 0) 0x7f68181d83d0 con 0x7f68180b4d60 2024-03-25T14:28:07.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.938+0000 7f680effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.938+0000 7f680effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.938+0000 7f680effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.938+0000 7f680effd640 10 monclient(hunting): my global_id is 4596 2024-03-25T14:28:07.941 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.938+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc004eb0 con 0x7f68180b4d60 2024-03-25T14:28:07.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.946+0000 7f680effd640 10 client.?.objecter ms_handle_connect 0x7f681814be90 2024-03-25T14:28:07.949 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.946+0000 7f680effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:07.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.946+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1077345427 0 0) 0x7f67fc004eb0 con 0x7f68180b4d60 2024-03-25T14:28:07.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.946+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68181d83d0 con 0x7f68180b4d60 2024-03-25T14:28:07.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.946+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6808001bc0 con 0x7f68180b4d60 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.946+0000 7f680effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.947+0000 7f680effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.947+0000 7f680effd640 10 monclient(hunting): dump: 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:07.951 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.947+0000 7f680effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:07.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.957+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944387484 0 0) 0x7f68181d77a0 con 0x7f68180b1730 2024-03-25T14:28:07.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.958+0000 7f680effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:07.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.958+0000 7f680effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:07.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.958+0000 7f680effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:07.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.958+0000 7f680effd640 10 monclient(hunting): my global_id is 4600 2024-03-25T14:28:07.961 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.958+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc004eb0 con 0x7f68180b1730 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3629216711 0 0) 0x7f68181d83d0 con 0x7f68180b4d60 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 >> v1:172.21.15.63:6790/0 conn(0x7f681814be90 legacy=0x7f6818142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 >> v1:172.21.15.153:6789/0 conn(0x7f68180b1730 legacy=0x7f68181d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 1 monclient: found mon.a 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:07.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6818140d00 con 0x7f68180b4d60 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 10 monclient: _finish_auth 0 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f680effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 5 monclient: authenticate success, global_id 4596 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 10 monclient: _renew_subs 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6818138a80 con 0x7f68180b4d60 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 10 monclient: _renew_subs 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6818143640 con 0x7f68180b4d60 2024-03-25T14:28:07.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.969+0000 7f68202c2640 1 librados: init done 2024-03-25T14:28:07.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.970+0000 7f68202c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:07.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.970+0000 7f68202c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:07.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.970+0000 7f68202c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:07.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:07.970+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6818140ef0 con 0x7f68180b4d60 2024-03-25T14:28:08.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.291+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68080011f0 con 0x7f68180b4d60 2024-03-25T14:28:08.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.291+0000 7f680effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:08.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.291+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6808003960 con 0x7f68180b4d60 2024-03-25T14:28:08.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.291+0000 7f680effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:08.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.291+0000 7f680effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.291+0000 7f680effd640 10 monclient: dump: 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:08.296 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:08.297 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:08.297 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:08.297 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:08.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.291+0000 7f681c85b640 4 set_mon_vals no callback set 2024-03-25T14:28:08.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.292+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f6808003b80 con 0x7f68180b4d60 2024-03-25T14:28:08.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.292+0000 7f681e05e640 10 monclient: get_auth_request con 0x7f67fc06dd10 auth_method 0 2024-03-25T14:28:08.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.293+0000 7f680effd640 10 client.4596.objecter ms_handle_connect 0x7f67fc06dd10 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f6808054780 con 0x7f68180b4d60 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 10 client.4596.objecter ms_dispatch 0x7f6818000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6808050290 con 0x7f68180b4d60 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:08.303 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.299+0000 7f680effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:08.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.529+0000 7f68202c2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:08.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.529+0000 7f68202c2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:08.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.529+0000 7f68202c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:08.533 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.529+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68180afbd0 con 0x7f68180b4d60 2024-03-25T14:28:08.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.551+0000 7f680effd640 1 -- 172.21.15.63:0/3813760559 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f68080521c0 con 0x7f68180b4d60 2024-03-25T14:28:08.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.551+0000 7f680effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:08.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.551+0000 7f680effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:08.555 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:08.555 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:08.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 10 librados: watch_flush enter 2024-03-25T14:28:08.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 10 librados: watch_flush exit 2024-03-25T14:28:08.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T14:28:08.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 10 client.4596.objecter successfully canceled tick 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 >> v1:172.21.15.153:6800/1688255552 conn(0x7f67fc06dd10 legacy=0x7f67fc0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 10 monclient: shutdown 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 >> v1:172.21.15.63:6789/0 conn(0x7f68180b4d60 legacy=0x7f68181405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 shutdown_connections 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 >> 172.21.15.63:0/3813760559 conn(0x7f68180617a0 msgr2=0x7f6818061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 shutdown_connections 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.553+0000 7f68202c2640 1 -- 172.21.15.63:0/3813760559 wait complete. 2024-03-25T14:28:08.558 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:08.554+0000 7f68202c2640 1 librados: shutdown 2024-03-25T14:28:08.569 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:15.376 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:15.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.489+0000 7f0f48990640 10 monclient: get_monmap_and_config 2024-03-25T14:28:15.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.489+0000 7f0f48990640 10 monclient: build_initial_monmap 2024-03-25T14:28:15.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.489+0000 7f0f48990640 10 monclient: monmap: 2024-03-25T14:28:15.494 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:15.494 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:15.494 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:15.490877+0000 2024-03-25T14:28:15.494 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:15.490877+0000 2024-03-25T14:28:15.495 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:15.495 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:15.495 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:15.495 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:15.495 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:15.495 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.496 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.490+0000 7f0f48990640 1 Processor -- start 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.490+0000 7f0f48990640 1 -- start start 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.490+0000 7f0f48990640 10 monclient: init 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 10 monclient(hunting): picked mon.noname-c con 0x7f0f440b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 10 monclient(hunting): picked mon.noname-b con 0x7f0f440b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 10 monclient(hunting): picked mon.noname-a con 0x7f0f4414be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f440af940 con 0x7f0f4414be90 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f440b0a70 con 0x7f0f440b4d60 2024-03-25T14:28:15.497 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f440b0d50 con 0x7f0f440b1730 2024-03-25T14:28:15.498 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:15.498 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.491+0000 7f0f48990640 10 monclient(hunting): authenticate will time out at 1858.636413s 2024-03-25T14:28:15.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.499+0000 7f0f42d9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f0f4414be90 0x7f0f4414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49152/0 (socket says 172.21.15.63:49152) 2024-03-25T14:28:15.503 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.499+0000 7f0f42d9d640 1 -- 172.21.15.63:0/4277329654 learned_addr learned my addr 172.21.15.63:0/4277329654 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:15.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.508+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500514301 0 0) 0x7f0f440b0a70 con 0x7f0f440b4d60 2024-03-25T14:28:15.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.508+0000 7f0f4159a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.508+0000 7f0f4159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.508+0000 7f0f4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.508+0000 7f0f4159a640 10 monclient(hunting): my global_id is 4603 2024-03-25T14:28:15.513 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.508+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f2c003520 con 0x7f0f440b4d60 2024-03-25T14:28:15.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.519+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256206935 0 0) 0x7f0f440b0d50 con 0x7f0f440b1730 2024-03-25T14:28:15.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.519+0000 7f0f4159a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.519+0000 7f0f4159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.524 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.519+0000 7f0f4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.519+0000 7f0f4159a640 10 monclient(hunting): my global_id is 4580 2024-03-25T14:28:15.525 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.519+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f440b0a70 con 0x7f0f440b1730 2024-03-25T14:28:15.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.520+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1300224183 0 0) 0x7f0f2c003520 con 0x7f0f440b4d60 2024-03-25T14:28:15.526 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.520+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 >> v1:172.21.15.63:6790/0 conn(0x7f0f440b1730 legacy=0x7f0f440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.520+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 >> v1:172.21.15.63:6789/0 conn(0x7f0f4414be90 legacy=0x7f0f4414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.520+0000 7f0f4159a640 10 monclient: _finish_hunting 0 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.520+0000 7f0f4159a640 1 monclient: found mon.noname-c 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.520+0000 7f0f4159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.520+0000 7f0f4159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f440b1030 con 0x7f0f440b4d60 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 10 monclient: _finish_auth 0 2024-03-25T14:28:15.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 10 monclient: _check_auth_tickets 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f34001690 con 0x7f0f440b4d60 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f4159a640 10 monclient: dump: 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.528 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f48990640 5 monclient: authenticate success, global_id 4603 2024-03-25T14:28:15.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.521+0000 7f0f48990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:15.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f340017f0 con 0x7f0f440b4d60 2024-03-25T14:28:15.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f4159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:15.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f4159a640 1 -- 172.21.15.63:0/4277329654 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f34001c00 con 0x7f0f440b4d60 2024-03-25T14:28:15.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f4159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:15.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f4159a640 10 monclient: dump: 2024-03-25T14:28:15.541 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f48990640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:15.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.536+0000 7f0f48990640 4 set_mon_vals no callback set 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.537+0000 7f0f48990640 10 monclient: shutdown 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.537+0000 7f0f48990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.537+0000 7f0f48990640 1 -- 172.21.15.63:0/4277329654 >> v1:172.21.15.153:6789/0 conn(0x7f0f440b4d60 legacy=0x7f0f4413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.537+0000 7f0f48990640 1 -- 172.21.15.63:0/4277329654 shutdown_connections 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.537+0000 7f0f48990640 1 -- 172.21.15.63:0/4277329654 >> 172.21.15.63:0/4277329654 conn(0x7f0f440617a0 msgr2=0x7f0f44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.537+0000 7f0f48990640 1 -- 172.21.15.63:0/4277329654 shutdown_connections 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.537+0000 7f0f48990640 1 -- 172.21.15.63:0/4277329654 wait complete. 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.538+0000 7f0f48990640 10 monclient: build_initial_monmap 2024-03-25T14:28:15.545 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.538+0000 7f0f48990640 10 monclient: monmap: 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.538+0000 7f0f48990640 1 librados: starting msgr at 2024-03-25T14:28:15.546 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.538+0000 7f0f48990640 1 librados: starting objecter 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 1 Processor -- start 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 1 -- start start 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 1 librados: setting wanted keys 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 1 librados: calling monclient init 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient: init 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient(hunting): picked mon.noname-c con 0x7f0f440b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:15.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient(hunting): picked mon.noname-a con 0x7f0f440b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:15.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4414be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:15.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f44141f40 con 0x7f0f440b4d60 2024-03-25T14:28:15.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f44142360 con 0x7f0f4414be90 2024-03-25T14:28:15.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f441424f0 con 0x7f0f440b1730 2024-03-25T14:28:15.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:15.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.539+0000 7f0f48990640 10 monclient(hunting): authenticate will time out at 1858.684455s 2024-03-25T14:28:15.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.559+0000 7f0f42d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0f4414be90 0x7f0f441e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:60014/0 (socket says 172.21.15.63:60014) 2024-03-25T14:28:15.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.559+0000 7f0f42d9d640 1 -- 172.21.15.63:0/3981728642 learned_addr learned my addr 172.21.15.63:0/3981728642 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:15.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.576+0000 7f0f32ffd640 10 client.?.objecter ms_handle_connect 0x7f0f4414be90 2024-03-25T14:28:15.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.576+0000 7f0f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:15.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.576+0000 7f0f32ffd640 10 client.?.objecter ms_handle_connect 0x7f0f440b4d60 2024-03-25T14:28:15.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.576+0000 7f0f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:15.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.576+0000 7f0f32ffd640 10 client.?.objecter ms_handle_connect 0x7f0f440b1730 2024-03-25T14:28:15.581 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.576+0000 7f0f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:15.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.585+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725192636 0 0) 0x7f0f44142360 con 0x7f0f4414be90 2024-03-25T14:28:15.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.585+0000 7f0f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.585+0000 7f0f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.585+0000 7f0f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.585+0000 7f0f32ffd640 10 monclient(hunting): my global_id is 4606 2024-03-25T14:28:15.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.585+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f24004eb0 con 0x7f0f4414be90 2024-03-25T14:28:15.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.591+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161326735 0 0) 0x7f0f44141f40 con 0x7f0f440b4d60 2024-03-25T14:28:15.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.591+0000 7f0f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.591+0000 7f0f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.591+0000 7f0f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.591+0000 7f0f32ffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T14:28:15.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.591+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f44142360 con 0x7f0f440b4d60 2024-03-25T14:28:15.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.593+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719669774 0 0) 0x7f0f440b1030 con 0x7f0f440b1730 2024-03-25T14:28:15.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.593+0000 7f0f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:15.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.593+0000 7f0f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:15.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.593+0000 7f0f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:15.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.593+0000 7f0f32ffd640 10 monclient(hunting): my global_id is 4583 2024-03-25T14:28:15.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.593+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f44141f40 con 0x7f0f440b1730 2024-03-25T14:28:15.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.601+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2019656020 0 0) 0x7f0f24004eb0 con 0x7f0f4414be90 2024-03-25T14:28:15.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.601+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f440b1030 con 0x7f0f4414be90 2024-03-25T14:28:15.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.601+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f3c001b10 con 0x7f0f4414be90 2024-03-25T14:28:15.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.601+0000 7f0f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.601+0000 7f0f32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:15.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.601+0000 7f0f32ffd640 10 monclient(hunting): dump: 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.602+0000 7f0f32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:15.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.628+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3295683505 0 0) 0x7f0f44141f40 con 0x7f0f440b1730 2024-03-25T14:28:15.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.628+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f24004eb0 con 0x7f0f440b1730 2024-03-25T14:28:15.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.628+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f34001960 con 0x7f0f440b1730 2024-03-25T14:28:15.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.628+0000 7f0f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.628+0000 7f0f32ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:15.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.628+0000 7f0f32ffd640 10 monclient(hunting): dump: 2024-03-25T14:28:15.634 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:15.634 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:15.634 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.635 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.635 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.635 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:15.635 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:15.635 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:15.635 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:15.636 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.629+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172072311 0 0) 0x7f0f44142360 con 0x7f0f440b4d60 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.629+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f44141f40 con 0x7f0f440b4d60 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.629+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f38001930 con 0x7f0f440b4d60 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.629+0000 7f0f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.629+0000 7f0f32ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.629+0000 7f0f32ffd640 10 monclient(hunting): dump: 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.638 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.639 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:15.639 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:15.639 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:15.639 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:15.639 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.636+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 988332910 0 0) 0x7f0f440b1030 con 0x7f0f4414be90 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 >> v1:172.21.15.63:6790/0 conn(0x7f0f440b1730 legacy=0x7f0f441405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 >> v1:172.21.15.63:6789/0 conn(0x7f0f440b4d60 legacy=0x7f0f441d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 1 monclient: found mon.b 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f44140d00 con 0x7f0f4414be90 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 10 monclient: _finish_auth 0 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 5 monclient: authenticate success, global_id 4606 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 10 monclient: _renew_subs 2024-03-25T14:28:15.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f44138150 con 0x7f0f4414be90 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 10 monclient: _renew_subs 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f441d8520 con 0x7f0f4414be90 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.637+0000 7f0f48990640 1 librados: init done 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.638+0000 7f0f48990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.638+0000 7f0f48990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.638+0000 7f0f48990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:15.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.638+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f44140f50 con 0x7f0f4414be90 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.939+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f3c002000 con 0x7f0f4414be90 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.939+0000 7f0f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.939+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f3c004d00 con 0x7f0f4414be90 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.939+0000 7f0f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.939+0000 7f0f32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.939+0000 7f0f32ffd640 10 monclient: dump: 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:15.944 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:15.945 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:15.945 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:15.945 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:15.945 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:15.945 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:15.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.939+0000 7f0f40d99640 4 set_mon_vals no callback set 2024-03-25T14:28:15.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.940+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f0f3c004f20 con 0x7f0f4414be90 2024-03-25T14:28:15.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.941+0000 7f0f4259c640 10 monclient: get_auth_request con 0x7f0f2406e0f0 auth_method 0 2024-03-25T14:28:15.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.941+0000 7f0f32ffd640 10 client.4606.objecter ms_handle_connect 0x7f0f2406e0f0 2024-03-25T14:28:15.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.943+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f0f3c054990 con 0x7f0f4414be90 2024-03-25T14:28:15.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.943+0000 7f0f32ffd640 10 client.4606.objecter ms_dispatch 0x7f0f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:15.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.943+0000 7f0f32ffd640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:15.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.943+0000 7f0f32ffd640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:15.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.943+0000 7f0f32ffd640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T14:28:15.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.946+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f3c0504a0 con 0x7f0f4414be90 2024-03-25T14:28:15.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.946+0000 7f0f32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:15.950 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:15.946+0000 7f0f32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:16.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.205+0000 7f0f48990640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:16.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.205+0000 7f0f48990640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:16.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.205+0000 7f0f48990640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:16.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.205+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f440627b0 con 0x7f0f4414be90 2024-03-25T14:28:16.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.236+0000 7f0f32ffd640 1 -- 172.21.15.63:0/3981728642 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f0f3c0523d0 con 0x7f0f4414be90 2024-03-25T14:28:16.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.236+0000 7f0f32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:16.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.236+0000 7f0f32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:16.241 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:16.241 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.238+0000 7f0f48990640 10 librados: watch_flush enter 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.238+0000 7f0f48990640 10 librados: watch_flush exit 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.238+0000 7f0f48990640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.238+0000 7f0f48990640 10 client.4606.objecter successfully canceled tick 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 >> v1:172.21.15.153:6800/1688255552 conn(0x7f0f2406e0f0 legacy=0x7f0f240705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 10 monclient: shutdown 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 >> v1:172.21.15.153:6789/0 conn(0x7f0f4414be90 legacy=0x7f0f441e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 shutdown_connections 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 >> 172.21.15.63:0/3981728642 conn(0x7f0f440617a0 msgr2=0x7f0f440b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:16.244 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 shutdown_connections 2024-03-25T14:28:16.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 1 -- 172.21.15.63:0/3981728642 wait complete. 2024-03-25T14:28:16.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:16.239+0000 7f0f48990640 1 librados: shutdown 2024-03-25T14:28:16.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:23.062 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.175+0000 7fbf4e71c640 10 monclient: get_monmap_and_config 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.175+0000 7fbf4e71c640 10 monclient: build_initial_monmap 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.175+0000 7fbf4e71c640 10 monclient: monmap: 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:23.176659+0000 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:23.176659+0000 2024-03-25T14:28:23.180 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:23.181 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:23.181 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:23.181 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:23.181 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:23.181 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:23.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.176+0000 7fbf4e71c640 1 Processor -- start 2024-03-25T14:28:23.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.176+0000 7fbf4e71c640 1 -- start start 2024-03-25T14:28:23.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.176+0000 7fbf4e71c640 10 monclient: init 2024-03-25T14:28:23.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:23.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:28:23.182 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 10 monclient(hunting): picked mon.noname-a con 0x7fbf480a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 10 monclient(hunting): picked mon.noname-c con 0x7fbf4813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 10 monclient(hunting): picked mon.noname-b con 0x7fbf481430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf48142520 con 0x7fbf480a98f0 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480b28a0 con 0x7fbf4813f710 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480b2a30 con 0x7fbf481430c0 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:23.183 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.177+0000 7fbf4e71c640 10 monclient(hunting): authenticate will time out at 1866.321827s 2024-03-25T14:28:23.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.182+0000 7fbf477fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbf4813f710 0x7fbf481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:45614/0 (socket says 172.21.15.63:45614) 2024-03-25T14:28:23.187 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.182+0000 7fbf477fe640 1 -- 172.21.15.63:0/3835349616 learned_addr learned my addr 172.21.15.63:0/3835349616 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:23.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.195+0000 7fbf47fff640 1 -- 172.21.15.63:0/3835349616 >> v1:172.21.15.63:6789/0 conn(0x7fbf480a98f0 legacy=0x7fbf480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T14:28:23.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.195+0000 7fbf47fff640 1 -- 172.21.15.63:0/3835349616 >> v1:172.21.15.63:6789/0 conn(0x7fbf480a98f0 legacy=0x7fbf480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T14:28:23.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.195+0000 7fbf47fff640 1 --1- 172.21.15.63:0/3835349616 >> v1:172.21.15.63:6789/0 conn(0x7fbf480a98f0 0x7fbf480a9cd0 :-1 s=OPENED pgs=61714 cs=1 l=1).handle_message read tag failed 2024-03-25T14:28:23.199 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.195+0000 7fbf47fff640 1 --1- 172.21.15.63:0/3835349616 >> v1:172.21.15.63:6789/0 conn(0x7fbf480a98f0 0x7fbf480a9cd0 :-1 s=OPENED pgs=61714 cs=1 l=1).fault on lossy channel, failing 2024-03-25T14:28:23.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.195+0000 7fbf46ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6789/0 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.196+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176181298 0 0) 0x7fbf480b28a0 con 0x7fbf4813f710 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.196+0000 7fbf46ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.196+0000 7fbf46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.196+0000 7fbf46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.196+0000 7fbf46ffd640 10 monclient(hunting): my global_id is 4609 2024-03-25T14:28:23.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.197+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf30003520 con 0x7fbf4813f710 2024-03-25T14:28:23.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1403956924 0 0) 0x7fbf30003520 con 0x7fbf4813f710 2024-03-25T14:28:23.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 >> v1:172.21.15.63:6790/0 conn(0x7fbf481430c0 legacy=0x7fbf4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 >> v1:172.21.15.63:6789/0 conn(0x7fbf480a98f0 legacy=0x7fbf480a9cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T14:28:23.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:23.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 1 monclient: found mon.noname-c 2024-03-25T14:28:23.210 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480b44c0 con 0x7fbf4813f710 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 10 monclient: _finish_auth 0 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbf3c001690 con 0x7fbf4813f710 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf46ffd640 10 monclient: dump: 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.211 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf4e71c640 5 monclient: authenticate success, global_id 4609 2024-03-25T14:28:23.212 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.205+0000 7fbf4e71c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf3c0017f0 con 0x7fbf4813f710 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf46ffd640 1 -- 172.21.15.63:0/3835349616 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbf3c001c40 con 0x7fbf4813f710 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf46ffd640 10 monclient: dump: 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.224 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf4e71c640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:23.225 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.219+0000 7fbf4e71c640 4 set_mon_vals no callback set 2024-03-25T14:28:23.227 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.220+0000 7fbf4e71c640 10 monclient: shutdown 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.220+0000 7fbf4e71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.220+0000 7fbf4e71c640 1 -- 172.21.15.63:0/3835349616 >> v1:172.21.15.153:6789/0 conn(0x7fbf4813f710 legacy=0x7fbf481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.220+0000 7fbf4e71c640 1 -- 172.21.15.63:0/3835349616 shutdown_connections 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.220+0000 7fbf4e71c640 1 -- 172.21.15.63:0/3835349616 >> 172.21.15.63:0/3835349616 conn(0x7fbf480b7ea0 msgr2=0x7fbf480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.220+0000 7fbf4e71c640 1 -- 172.21.15.63:0/3835349616 shutdown_connections 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.220+0000 7fbf4e71c640 1 -- 172.21.15.63:0/3835349616 wait complete. 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 10 monclient: build_initial_monmap 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 10 monclient: monmap: 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:23.228 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 1 librados: starting msgr at 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 1 librados: starting objecter 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 1 Processor -- start 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 1 -- start start 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 1 librados: setting wanted keys 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 1 librados: calling monclient init 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.221+0000 7fbf4e71c640 10 monclient: init 2024-03-25T14:28:23.229 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.222+0000 7fbf4e71c640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.222+0000 7fbf4e71c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.222+0000 7fbf4e71c640 10 monclient(hunting): picked mon.noname-c con 0x7fbf480a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.222+0000 7fbf4e71c640 10 monclient(hunting): picked mon.noname-b con 0x7fbf4813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.223+0000 7fbf4e71c640 10 monclient(hunting): picked mon.noname-a con 0x7fbf481430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.223+0000 7fbf4e71c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf481d7830 con 0x7fbf481430c0 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.223+0000 7fbf4e71c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf481d7500 con 0x7fbf4813f710 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.223+0000 7fbf4e71c640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf481d83d0 con 0x7fbf480a98f0 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.223+0000 7fbf4e71c640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:23.230 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.223+0000 7fbf4e71c640 10 monclient(hunting): authenticate will time out at 1866.367731s 2024-03-25T14:28:23.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.234+0000 7fbf4ccb9640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fbf481430c0 0x7fbf481e1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:46048/0 (socket says 172.21.15.63:46048) 2024-03-25T14:28:23.238 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.234+0000 7fbf4ccb9640 1 -- 172.21.15.63:0/2231320795 learned_addr learned my addr 172.21.15.63:0/2231320795 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:23.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.236+0000 7fbf44ff9640 10 client.?.objecter ms_handle_connect 0x7fbf481430c0 2024-03-25T14:28:23.241 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.236+0000 7fbf44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:23.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.241+0000 7fbf44ff9640 10 client.?.objecter ms_handle_connect 0x7fbf480a98f0 2024-03-25T14:28:23.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.241+0000 7fbf44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.246+0000 7fbf44ff9640 10 client.?.objecter ms_handle_connect 0x7fbf4813f710 2024-03-25T14:28:23.251 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.246+0000 7fbf44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599325467 0 0) 0x7fbf481d7830 con 0x7fbf481430c0 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): my global_id is 4608 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf34004eb0 con 0x7fbf481430c0 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449597175 0 0) 0x7fbf481d83d0 con 0x7fbf480a98f0 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:23.256 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 10 monclient(hunting): my global_id is 4589 2024-03-25T14:28:23.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.251+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf481d7830 con 0x7fbf480a98f0 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.345+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1307208897 0 0) 0x7fbf34004eb0 con 0x7fbf481430c0 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.345+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf481d83d0 con 0x7fbf481430c0 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.345+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbf40001c90 con 0x7fbf481430c0 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.345+0000 7fbf44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.345+0000 7fbf44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.345+0000 7fbf44ff9640 10 monclient(hunting): dump: 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:23.350 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:23.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.345+0000 7fbf44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 242270102 0 0) 0x7fbf481d83d0 con 0x7fbf481430c0 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 >> v1:172.21.15.63:6790/0 conn(0x7fbf480a98f0 legacy=0x7fbf48138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 >> v1:172.21.15.153:6789/0 conn(0x7fbf4813f710 legacy=0x7fbf481d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 1 monclient: found mon.a 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480abb90 con 0x7fbf481430c0 2024-03-25T14:28:23.382 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 10 monclient: _finish_auth 0 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 5 monclient: authenticate success, global_id 4608 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 10 monclient: _renew_subs 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf48137870 con 0x7fbf481430c0 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 10 monclient: _renew_subs 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf48135040 con 0x7fbf481430c0 2024-03-25T14:28:23.383 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.377+0000 7fbf4e71c640 1 librados: init done 2024-03-25T14:28:23.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.378+0000 7fbf4e71c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:23.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.378+0000 7fbf4e71c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:23.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.378+0000 7fbf4e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:23.384 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.378+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf480627b0 con 0x7fbf481430c0 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf40003990 con 0x7fbf481430c0 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fbf40003ce0 con 0x7fbf481430c0 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf44ff9640 10 monclient: dump: 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:23.732 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:23.733 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:23.733 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:23.733 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:23.733 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:23.733 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:23.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf467fc640 4 set_mon_vals no callback set 2024-03-25T14:28:23.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.727+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fbf40003f00 con 0x7fbf481430c0 2024-03-25T14:28:23.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.728+0000 7fbf47fff640 10 monclient: get_auth_request con 0x7fbf3406dd10 auth_method 0 2024-03-25T14:28:23.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.728+0000 7fbf44ff9640 10 client.4608.objecter ms_handle_connect 0x7fbf3406dd10 2024-03-25T14:28:23.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fbf40054ad0 con 0x7fbf481430c0 2024-03-25T14:28:23.738 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 10 client.4608.objecter ms_dispatch 0x7fbf48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:23.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 3 client.4608.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:23.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 3 client.4608.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:23.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-25T14:28:23.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf400505e0 con 0x7fbf481430c0 2024-03-25T14:28:23.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:23.739 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.734+0000 7fbf44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:24.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.995+0000 7fbf4e71c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:24.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.995+0000 7fbf4e71c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:24.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.995+0000 7fbf4e71c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:24.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:23.995+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf481352f0 con 0x7fbf481430c0 2024-03-25T14:28:24.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.041+0000 7fbf44ff9640 1 -- 172.21.15.63:0/2231320795 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fbf40052510 con 0x7fbf481430c0 2024-03-25T14:28:24.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.041+0000 7fbf44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:24.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.041+0000 7fbf44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:24.046 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:24.047 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:24.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 10 librados: watch_flush enter 2024-03-25T14:28:24.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 10 librados: watch_flush exit 2024-03-25T14:28:24.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-25T14:28:24.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 10 client.4608.objecter successfully canceled tick 2024-03-25T14:28:24.049 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 >> v1:172.21.15.153:6800/1688255552 conn(0x7fbf3406dd10 legacy=0x7fbf340701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 10 monclient: shutdown 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 >> v1:172.21.15.63:6789/0 conn(0x7fbf481430c0 legacy=0x7fbf481e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 shutdown_connections 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 >> 172.21.15.63:0/2231320795 conn(0x7fbf480b7ea0 msgr2=0x7fbf4813ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 shutdown_connections 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 1 -- 172.21.15.63:0/2231320795 wait complete. 2024-03-25T14:28:24.050 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:24.044+0000 7fbf4e71c640 1 librados: shutdown 2024-03-25T14:28:24.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:27.572 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:06.084 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.617 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:06.076 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.618 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:06.077 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.619 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:06.131 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.629 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:06.088 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.661 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:06.173 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.705 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:06.217 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.711 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:06.223 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:27.732 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:06.244 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:30.867 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:30.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.975+0000 7f162cb67640 10 monclient: get_monmap_and_config 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.975+0000 7f162cb67640 10 monclient: build_initial_monmap 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.975+0000 7f162cb67640 10 monclient: monmap: 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:30.976457+0000 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:30.976457+0000 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:30.980 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.976+0000 7f162cb67640 1 Processor -- start 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.976+0000 7f162cb67640 1 -- start start 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.976+0000 7f162cb67640 10 monclient: init 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.976+0000 7f162cb67640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.976+0000 7f162cb67640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.976+0000 7f162cb67640 10 monclient(hunting): picked mon.noname-a con 0x7f16280705b0 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.977+0000 7f162cb67640 10 monclient(hunting): picked mon.noname-c con 0x7f16281402d0 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.977+0000 7f162cb67640 10 monclient(hunting): picked mon.noname-b con 0x7f1628149ae0 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.977+0000 7f162cb67640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16280718c0 con 0x7f16280705b0 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.977+0000 7f162cb67640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16280714c0 con 0x7f16281402d0 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.977+0000 7f162cb67640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1628072b50 con 0x7f1628149ae0 2024-03-25T14:28:30.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.977+0000 7f162cb67640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:30.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.977+0000 7f162cb67640 10 monclient(hunting): authenticate will time out at 1874.121653s 2024-03-25T14:28:30.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.990+0000 7f1626d9d640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f1628149ae0 0x7f162814bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:41472/0 (socket says 172.21.15.63:41472) 2024-03-25T14:28:30.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:30.990+0000 7f1626d9d640 1 -- 172.21.15.63:0/3619870851 learned_addr learned my addr 172.21.15.63:0/3619870851 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:31.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.006+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066435672 0 0) 0x7f1628072b50 con 0x7f1628149ae0 2024-03-25T14:28:31.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.006+0000 7f162559a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:31.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.006+0000 7f162559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:31.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.006+0000 7f162559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:31.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.006+0000 7f162559a640 10 monclient(hunting): my global_id is 4592 2024-03-25T14:28:31.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.006+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1610003520 con 0x7f1628149ae0 2024-03-25T14:28:31.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.012+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461165686 0 0) 0x7f16280718c0 con 0x7f16280705b0 2024-03-25T14:28:31.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.012+0000 7f162559a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:31.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.012+0000 7f162559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:31.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.012+0000 7f162559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:31.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.012+0000 7f162559a640 10 monclient(hunting): my global_id is 4611 2024-03-25T14:28:31.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.012+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1628072b50 con 0x7f16280705b0 2024-03-25T14:28:31.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.017+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249449516 0 0) 0x7f16280714c0 con 0x7f16281402d0 2024-03-25T14:28:31.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.017+0000 7f162559a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:31.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.017+0000 7f162559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:31.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.017+0000 7f162559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:31.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.017+0000 7f162559a640 10 monclient(hunting): my global_id is 4615 2024-03-25T14:28:31.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.017+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16280718c0 con 0x7f16281402d0 2024-03-25T14:28:31.031 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 706506933 0 0) 0x7f1610003520 con 0x7f1628149ae0 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 >> v1:172.21.15.153:6789/0 conn(0x7f16281402d0 legacy=0x7f1628140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 >> v1:172.21.15.63:6789/0 conn(0x7f16280705b0 legacy=0x7f162813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 10 monclient: _finish_hunting 0 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 1 monclient: found mon.noname-b 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162813dd10 con 0x7f1628149ae0 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 10 monclient: _finish_auth 0 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 10 monclient: _check_auth_tickets 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:31.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1620001a10 con 0x7f1628149ae0 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162cb67640 5 monclient: authenticate success, global_id 4592 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162cb67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162559a640 10 monclient: dump: 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:31.033 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:31.034 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:31.034 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:31.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.026+0000 7f162cb67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1620001b70 con 0x7f1628149ae0 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162559a640 1 -- 172.21.15.63:0/3619870851 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1620001f80 con 0x7f1628149ae0 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162cb67640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162cb67640 4 set_mon_vals no callback set 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.046+0000 7f162559a640 10 monclient: dump: 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:31.051 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:31.052 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.047+0000 7f162cb67640 10 monclient: shutdown 2024-03-25T14:28:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.047+0000 7f162cb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:31.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.047+0000 7f162cb67640 1 -- 172.21.15.63:0/3619870851 >> v1:172.21.15.63:6790/0 conn(0x7f1628149ae0 legacy=0x7f162814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.047+0000 7f162cb67640 1 -- 172.21.15.63:0/3619870851 shutdown_connections 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.047+0000 7f162cb67640 1 -- 172.21.15.63:0/3619870851 >> 172.21.15.63:0/3619870851 conn(0x7f16280b5d10 msgr2=0x7f16280b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.047+0000 7f162cb67640 1 -- 172.21.15.63:0/3619870851 shutdown_connections 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.047+0000 7f162cb67640 1 -- 172.21.15.63:0/3619870851 wait complete. 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.048+0000 7f162cb67640 10 monclient: build_initial_monmap 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.048+0000 7f162cb67640 10 monclient: monmap: 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:31.054 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.049+0000 7f162cb67640 1 librados: starting msgr at 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.049+0000 7f162cb67640 1 librados: starting objecter 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.049+0000 7f162cb67640 1 Processor -- start 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.049+0000 7f162cb67640 1 -- start start 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.049+0000 7f162cb67640 1 librados: setting wanted keys 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.049+0000 7f162cb67640 1 librados: calling monclient init 2024-03-25T14:28:31.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.049+0000 7f162cb67640 10 monclient: init 2024-03-25T14:28:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 10 monclient(hunting): picked mon.noname-c con 0x7f16280705b0 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 10 monclient(hunting): picked mon.noname-a con 0x7f16281402d0 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 10 monclient(hunting): picked mon.noname-b con 0x7f1628149ae0 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16281434a0 con 0x7f16281402d0 2024-03-25T14:28:31.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16281dae70 con 0x7f1628149ae0 2024-03-25T14:28:31.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1628144250 con 0x7f16280705b0 2024-03-25T14:28:31.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:31.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.050+0000 7f162cb67640 10 monclient(hunting): authenticate will time out at 1874.194770s 2024-03-25T14:28:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.055+0000 7f1626d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1628149ae0 0x7f16281e31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:56538/0 (socket says 172.21.15.63:56538) 2024-03-25T14:28:31.060 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.055+0000 7f1626d9d640 1 -- 172.21.15.63:0/1185050708 learned_addr learned my addr 172.21.15.63:0/1185050708 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.056+0000 7f1616ffd640 10 client.?.objecter ms_handle_connect 0x7f1628149ae0 2024-03-25T14:28:31.061 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.056+0000 7f1616ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:31.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.059+0000 7f1616ffd640 10 client.?.objecter ms_handle_connect 0x7f16280705b0 2024-03-25T14:28:31.063 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.059+0000 7f1616ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:31.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.064+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549332208 0 0) 0x7f16281dae70 con 0x7f1628149ae0 2024-03-25T14:28:31.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.064+0000 7f1616ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:31.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.064+0000 7f1616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:31.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.064+0000 7f1616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:31.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.064+0000 7f1616ffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T14:28:31.069 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.064+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1608004eb0 con 0x7f1628149ae0 2024-03-25T14:28:31.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.066+0000 7f1616ffd640 10 client.?.objecter ms_handle_connect 0x7f16281402d0 2024-03-25T14:28:31.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.066+0000 7f1616ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:31.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.067+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953226987 0 0) 0x7f1628144250 con 0x7f16280705b0 2024-03-25T14:28:31.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.067+0000 7f1616ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:31.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.067+0000 7f1616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:31.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.067+0000 7f1616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:31.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.067+0000 7f1616ffd640 10 monclient(hunting): my global_id is 4595 2024-03-25T14:28:31.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.067+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16281dae70 con 0x7f16280705b0 2024-03-25T14:28:31.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.073+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2382136756 0 0) 0x7f1608004eb0 con 0x7f1628149ae0 2024-03-25T14:28:31.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.073+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1628144250 con 0x7f1628149ae0 2024-03-25T14:28:31.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.073+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1620001e80 con 0x7f1628149ae0 2024-03-25T14:28:31.077 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.073+0000 7f1616ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.073+0000 7f1616ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.073+0000 7f1616ffd640 10 monclient(hunting): dump: 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:31.078 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.073+0000 7f1616ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:31.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.074+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172822259 0 0) 0x7f16281434a0 con 0x7f16281402d0 2024-03-25T14:28:31.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.074+0000 7f1616ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:31.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.074+0000 7f1616ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:31.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.074+0000 7f1616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:31.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.074+0000 7f1616ffd640 10 monclient(hunting): my global_id is 4614 2024-03-25T14:28:31.079 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.074+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1608004eb0 con 0x7f16281402d0 2024-03-25T14:28:31.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3538286396 0 0) 0x7f1628144250 con 0x7f1628149ae0 2024-03-25T14:28:31.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 >> v1:172.21.15.63:6790/0 conn(0x7f16280705b0 legacy=0x7f1628141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 >> v1:172.21.15.63:6789/0 conn(0x7f16281402d0 legacy=0x7f16281db350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:31.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 1 monclient: found mon.b 2024-03-25T14:28:31.087 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162806da00 con 0x7f1628149ae0 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 10 monclient: _finish_auth 0 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f1616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 5 monclient: authenticate success, global_id 4618 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 10 monclient: _renew_subs 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162806dc50 con 0x7f1628149ae0 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 10 monclient: _renew_subs 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16281424e0 con 0x7f1628149ae0 2024-03-25T14:28:31.088 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.082+0000 7f162cb67640 1 librados: init done 2024-03-25T14:28:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.083+0000 7f162cb67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.083+0000 7f162cb67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.083+0000 7f162cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:31.089 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.083+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16280627b0 con 0x7f1628149ae0 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.102+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1620002080 con 0x7f1628149ae0 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.102+0000 7f1616ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.102+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f1620004fb0 con 0x7f1628149ae0 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.102+0000 7f1616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.102+0000 7f1616ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.102+0000 7f1616ffd640 10 monclient: dump: 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:31.107 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:31.108 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:31.108 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:31.108 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:31.108 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:31.108 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:31.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.102+0000 7f1624d99640 4 set_mon_vals no callback set 2024-03-25T14:28:31.108 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.103+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f162001af80 con 0x7f1628149ae0 2024-03-25T14:28:31.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.103+0000 7f162659c640 10 monclient: get_auth_request con 0x7f160806de00 auth_method 0 2024-03-25T14:28:31.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.104+0000 7f1616ffd640 10 client.4618.objecter ms_handle_connect 0x7f160806de00 2024-03-25T14:28:31.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.104+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f1620001560 con 0x7f1628149ae0 2024-03-25T14:28:31.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.104+0000 7f1616ffd640 10 client.4618.objecter ms_dispatch 0x7f1628000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:31.109 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.104+0000 7f1616ffd640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:31.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.104+0000 7f1616ffd640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:31.110 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.104+0000 7f1616ffd640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T14:28:31.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.355+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1620050410 con 0x7f1628149ae0 2024-03-25T14:28:31.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.355+0000 7f1616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:31.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.355+0000 7f1616ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:31.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.584+0000 7f162cb67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:31.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.584+0000 7f162cb67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:31.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.584+0000 7f162cb67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:31.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.584+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16281443e0 con 0x7f1628149ae0 2024-03-25T14:28:31.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.610+0000 7f1616ffd640 1 -- 172.21.15.63:0/1185050708 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f1620052510 con 0x7f1628149ae0 2024-03-25T14:28:31.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.610+0000 7f1616ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:31.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.610+0000 7f1616ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:31.616 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:31.616 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:31.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.613+0000 7f162cb67640 10 librados: watch_flush enter 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 10 librados: watch_flush exit 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 10 client.4618.objecter successfully canceled tick 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 >> v1:172.21.15.153:6800/1688255552 conn(0x7f160806de00 legacy=0x7f16080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 10 monclient: shutdown 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 >> v1:172.21.15.153:6789/0 conn(0x7f1628149ae0 legacy=0x7f16281e31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 shutdown_connections 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 >> 172.21.15.63:0/1185050708 conn(0x7f16280b5d10 msgr2=0x7f16280b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 shutdown_connections 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 1 -- 172.21.15.63:0/1185050708 wait complete. 2024-03-25T14:28:31.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:31.614+0000 7f162cb67640 1 librados: shutdown 2024-03-25T14:28:31.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:38.438 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:38.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.549+0000 7f0c44e4e640 10 monclient: get_monmap_and_config 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.549+0000 7f0c44e4e640 10 monclient: build_initial_monmap 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.549+0000 7f0c44e4e640 10 monclient: monmap: 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:38.550274+0000 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:38.550274+0000 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:38.554 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:38.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.549+0000 7f0c44e4e640 1 Processor -- start 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.549+0000 7f0c44e4e640 1 -- start start 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.549+0000 7f0c44e4e640 10 monclient: init 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f0c400a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f0c4013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f0c401430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c40142520 con 0x7f0c400a98f0 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c400b28a0 con 0x7f0c401430c0 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c400b2a30 con 0x7f0c4013f710 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:38.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.550+0000 7f0c44e4e640 10 monclient(hunting): authenticate will time out at 1881.695295s 2024-03-25T14:28:38.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.561+0000 7f0c3dd9b640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0c4013f710 0x7f0c401429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:46342/0 (socket says 172.21.15.63:46342) 2024-03-25T14:28:38.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.561+0000 7f0c3dd9b640 1 -- 172.21.15.63:0/85287844 learned_addr learned my addr 172.21.15.63:0/85287844 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:38.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.568+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979225431 0 0) 0x7f0c400b28a0 con 0x7f0c401430c0 2024-03-25T14:28:38.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.568+0000 7f0c3d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.568+0000 7f0c3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.568+0000 7f0c3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.568+0000 7f0c3d59a640 10 monclient(hunting): my global_id is 4621 2024-03-25T14:28:38.573 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.568+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c28003520 con 0x7f0c401430c0 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.569+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69747555 0 0) 0x7f0c400b2a30 con 0x7f0c4013f710 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.569+0000 7f0c3d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.569+0000 7f0c3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.569+0000 7f0c3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.569+0000 7f0c3d59a640 10 monclient(hunting): my global_id is 4598 2024-03-25T14:28:38.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.569+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c400b28a0 con 0x7f0c4013f710 2024-03-25T14:28:38.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393461232 0 0) 0x7f0c400b28a0 con 0x7f0c4013f710 2024-03-25T14:28:38.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 >> v1:172.21.15.153:6789/0 conn(0x7f0c401430c0 legacy=0x7f0c4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 >> v1:172.21.15.63:6789/0 conn(0x7f0c400a98f0 legacy=0x7f0c400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 10 monclient: _finish_hunting 0 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 1 monclient: found mon.noname-b 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c400b44c0 con 0x7f0c4013f710 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 10 monclient: _finish_auth 0 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 10 monclient: _check_auth_tickets 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:38.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0c34001670 con 0x7f0c4013f710 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c44e4e640 5 monclient: authenticate success, global_id 4598 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c44e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c3d59a640 10 monclient: dump: 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:38.587 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:38.588 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:38.588 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:38.588 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.580+0000 7f0c44e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:38.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c340017d0 con 0x7f0c4013f710 2024-03-25T14:28:38.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c3d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:38.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c3d59a640 1 -- 172.21.15.63:0/85287844 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0c34001ba0 con 0x7f0c4013f710 2024-03-25T14:28:38.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c3d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:38.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c3d59a640 10 monclient: dump: 2024-03-25T14:28:38.596 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c44e4e640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:38.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.591+0000 7f0c44e4e640 4 set_mon_vals no callback set 2024-03-25T14:28:38.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.592+0000 7f0c44e4e640 10 monclient: shutdown 2024-03-25T14:28:38.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.592+0000 7f0c44e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:38.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.592+0000 7f0c44e4e640 1 -- 172.21.15.63:0/85287844 >> v1:172.21.15.63:6790/0 conn(0x7f0c4013f710 legacy=0x7f0c401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.592+0000 7f0c44e4e640 1 -- 172.21.15.63:0/85287844 shutdown_connections 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.592+0000 7f0c44e4e640 1 -- 172.21.15.63:0/85287844 >> 172.21.15.63:0/85287844 conn(0x7f0c400b7ea0 msgr2=0x7f0c400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.592+0000 7f0c44e4e640 1 -- 172.21.15.63:0/85287844 shutdown_connections 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.592+0000 7f0c44e4e640 1 -- 172.21.15.63:0/85287844 wait complete. 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.593+0000 7f0c44e4e640 10 monclient: build_initial_monmap 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.593+0000 7f0c44e4e640 10 monclient: monmap: 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:38.600 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 librados: starting msgr at 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 librados: starting objecter 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 Processor -- start 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 -- start start 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 librados: setting wanted keys 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 librados: calling monclient init 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient: init 2024-03-25T14:28:38.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f0c400a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f0c4013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f0c401430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c401d78e0 con 0x7f0c400a98f0 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c401d7500 con 0x7f0c401430c0 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c401d7690 con 0x7f0c4013f710 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:38.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.594+0000 7f0c44e4e640 10 monclient(hunting): authenticate will time out at 1881.739490s 2024-03-25T14:28:38.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.601+0000 7f0c3e59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f0c400a98f0 0x7f0c40134de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:49154/0 (socket says 172.21.15.63:49154) 2024-03-25T14:28:38.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.601+0000 7f0c3e59c640 1 -- 172.21.15.63:0/2339327840 learned_addr learned my addr 172.21.15.63:0/2339327840 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:38.609 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.604+0000 7f0c2effd640 10 client.?.objecter ms_handle_connect 0x7f0c400a98f0 2024-03-25T14:28:38.609 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.604+0000 7f0c2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:38.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.611+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705367960 0 0) 0x7f0c401d78e0 con 0x7f0c400a98f0 2024-03-25T14:28:38.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.611+0000 7f0c2effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.611+0000 7f0c2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.611+0000 7f0c2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.611+0000 7f0c2effd640 10 monclient(hunting): my global_id is 4620 2024-03-25T14:28:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.611+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c20004eb0 con 0x7f0c400a98f0 2024-03-25T14:28:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.612+0000 7f0c2effd640 10 client.?.objecter ms_handle_connect 0x7f0c4013f710 2024-03-25T14:28:38.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.612+0000 7f0c2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:38.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.614+0000 7f0c2effd640 10 client.?.objecter ms_handle_connect 0x7f0c401430c0 2024-03-25T14:28:38.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.614+0000 7f0c2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1108672362 0 0) 0x7f0c20004eb0 con 0x7f0c400a98f0 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c401d78e0 con 0x7f0c400a98f0 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0c30001a30 con 0x7f0c400a98f0 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): dump: 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:38.631 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862009316 0 0) 0x7f0c401d7690 con 0x7f0c4013f710 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 10 monclient(hunting): my global_id is 4601 2024-03-25T14:28:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.626+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c20004eb0 con 0x7f0c4013f710 2024-03-25T14:28:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.628+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622866051 0 0) 0x7f0c401d7500 con 0x7f0c401430c0 2024-03-25T14:28:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.628+0000 7f0c2effd640 10 monclient(hunting): _negotiate 2024-03-25T14:28:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.628+0000 7f0c2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.628+0000 7f0c2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:38.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.628+0000 7f0c2effd640 10 monclient(hunting): my global_id is 4624 2024-03-25T14:28:38.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.628+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c401d7690 con 0x7f0c401430c0 2024-03-25T14:28:38.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1600143215 0 0) 0x7f0c401d78e0 con 0x7f0c400a98f0 2024-03-25T14:28:38.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 >> v1:172.21.15.63:6790/0 conn(0x7f0c4013f710 legacy=0x7f0c401d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 >> v1:172.21.15.153:6789/0 conn(0x7f0c401430c0 legacy=0x7f0c40138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:38.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 10 monclient: _finish_hunting 0 2024-03-25T14:28:38.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 1 monclient: found mon.a 2024-03-25T14:28:38.649 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c401377c0 con 0x7f0c400a98f0 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 10 monclient: _finish_auth 0 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 10 monclient: _check_auth_tickets 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 5 monclient: authenticate success, global_id 4620 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 10 monclient: _renew_subs 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c40137a70 con 0x7f0c400a98f0 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 10 monclient: _renew_subs 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c401341f0 con 0x7f0c400a98f0 2024-03-25T14:28:38.650 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.644+0000 7f0c44e4e640 1 librados: init done 2024-03-25T14:28:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.645+0000 7f0c44e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.645+0000 7f0c44e4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.645+0000 7f0c44e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:38.651 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.645+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c400abbe0 con 0x7f0c400a98f0 2024-03-25T14:28:38.905 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.901+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c30001cb0 con 0x7f0c400a98f0 2024-03-25T14:28:38.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.901+0000 7f0c2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:38.906 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.901+0000 7f0c3cd99640 4 set_mon_vals no callback set 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.971+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0c30003a60 con 0x7f0c400a98f0 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.971+0000 7f0c2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.971+0000 7f0c2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.971+0000 7f0c2effd640 10 monclient: dump: 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:38.976 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:38.977 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:38.977 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:38.977 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:38.977 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:38.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.972+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f0c30003c40 con 0x7f0c400a98f0 2024-03-25T14:28:38.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.973+0000 7f0c3dd9b640 10 monclient: get_auth_request con 0x7f0c2006de80 auth_method 0 2024-03-25T14:28:38.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.973+0000 7f0c2effd640 10 client.4620.objecter ms_handle_connect 0x7f0c2006de80 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f0c30054950 con 0x7f0c400a98f0 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 10 client.4620.objecter ms_dispatch 0x7f0c40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 3 client.4620.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 3 client.4620.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c30050290 con 0x7f0c400a98f0 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:38.987 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:38.982+0000 7f0c2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:39.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.240+0000 7f0c44e4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:39.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.240+0000 7f0c44e4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:39.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.240+0000 7f0c44e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:39.245 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.240+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c400627b0 con 0x7f0c400a98f0 2024-03-25T14:28:39.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.277+0000 7f0c2effd640 1 -- 172.21.15.63:0/2339327840 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f0c30052390 con 0x7f0c400a98f0 2024-03-25T14:28:39.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.277+0000 7f0c2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:39.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.277+0000 7f0c2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:39.282 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:39.283 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:39.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.279+0000 7f0c44e4e640 10 librados: watch_flush enter 2024-03-25T14:28:39.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.279+0000 7f0c44e4e640 10 librados: watch_flush exit 2024-03-25T14:28:39.285 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.279+0000 7f0c44e4e640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.279+0000 7f0c44e4e640 10 client.4620.objecter successfully canceled tick 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 >> v1:172.21.15.153:6800/1688255552 conn(0x7f0c2006de80 legacy=0x7f0c20070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 10 monclient: shutdown 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 >> v1:172.21.15.63:6789/0 conn(0x7f0c400a98f0 legacy=0x7f0c40134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 shutdown_connections 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 >> 172.21.15.63:0/2339327840 conn(0x7f0c400b7ea0 msgr2=0x7f0c40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 shutdown_connections 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 1 -- 172.21.15.63:0/2339327840 wait complete. 2024-03-25T14:28:39.286 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:39.280+0000 7f0c44e4e640 1 librados: shutdown 2024-03-25T14:28:39.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:46.100 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.210+0000 7fdcabc2e640 10 monclient: get_monmap_and_config 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.210+0000 7fdcabc2e640 10 monclient: build_initial_monmap 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.210+0000 7fdcabc2e640 10 monclient: monmap: 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:46.211571+0000 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:46.211571+0000 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:46.215 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:46.216 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 1 Processor -- start 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 1 -- start start 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient: init 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient(hunting): picked mon.noname-c con 0x7fdca40b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient(hunting): picked mon.noname-b con 0x7fdca40b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient(hunting): picked mon.noname-a con 0x7fdca414be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40af940 con 0x7fdca414be90 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40b0a70 con 0x7fdca40b4d60 2024-03-25T14:28:46.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40b0d50 con 0x7fdca40b1730 2024-03-25T14:28:46.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:46.218 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.211+0000 7fdcabc2e640 10 monclient(hunting): authenticate will time out at 1889.356500s 2024-03-25T14:28:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.222+0000 7fdca91c9640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fdca40b1730 0x7fdca40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:50064/0 (socket says 172.21.15.63:50064) 2024-03-25T14:28:46.226 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.222+0000 7fdca91c9640 1 -- 172.21.15.63:0/3934440334 learned_addr learned my addr 172.21.15.63:0/3934440334 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:46.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.234+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253555591 0 0) 0x7fdca40b0d50 con 0x7fdca40b1730 2024-03-25T14:28:46.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.234+0000 7fdca89c8640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.234+0000 7fdca89c8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.234+0000 7fdca89c8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.234+0000 7fdca89c8640 10 monclient(hunting): my global_id is 4604 2024-03-25T14:28:46.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.234+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc90003520 con 0x7fdca40b1730 2024-03-25T14:28:46.242 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.235+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649618479 0 0) 0x7fdca40af940 con 0x7fdca414be90 2024-03-25T14:28:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.235+0000 7fdca89c8640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.235+0000 7fdca89c8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.235+0000 7fdca89c8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.235+0000 7fdca89c8640 10 monclient(hunting): my global_id is 4623 2024-03-25T14:28:46.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.235+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca40b0d50 con 0x7fdca414be90 2024-03-25T14:28:46.252 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908376668 0 0) 0x7fdca40b0a70 con 0x7fdca40b4d60 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient(hunting): my global_id is 4627 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca40af940 con 0x7fdca40b4d60 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231107413 0 0) 0x7fdca40b0d50 con 0x7fdca414be90 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 >> v1:172.21.15.63:6790/0 conn(0x7fdca40b1730 legacy=0x7fdca40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 >> v1:172.21.15.153:6789/0 conn(0x7fdca40b4d60 legacy=0x7fdca413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient: _finish_hunting 0 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 monclient: found mon.noname-a 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:46.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40b1030 con 0x7fdca414be90 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient: _finish_auth 0 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient: _check_auth_tickets 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdc9c0019f0 con 0x7fdca414be90 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdca89c8640 10 monclient: dump: 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.254 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.255 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:46.255 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:46.255 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:46.255 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:46.255 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:46.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdcabc2e640 5 monclient: authenticate success, global_id 4623 2024-03-25T14:28:46.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.247+0000 7fdcabc2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:46.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc9c001b50 con 0x7fdca414be90 2024-03-25T14:28:46.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdca89c8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:46.262 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdca89c8640 1 -- 172.21.15.63:0/3934440334 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdc9c001f60 con 0x7fdca414be90 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdca89c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdca89c8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdca89c8640 10 monclient: dump: 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:46.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdcabc2e640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:46.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.258+0000 7fdcabc2e640 4 set_mon_vals no callback set 2024-03-25T14:28:46.265 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.259+0000 7fdcabc2e640 10 monclient: shutdown 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.259+0000 7fdcabc2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.259+0000 7fdcabc2e640 1 -- 172.21.15.63:0/3934440334 >> v1:172.21.15.63:6789/0 conn(0x7fdca414be90 legacy=0x7fdca414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.259+0000 7fdcabc2e640 1 -- 172.21.15.63:0/3934440334 shutdown_connections 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.259+0000 7fdcabc2e640 1 -- 172.21.15.63:0/3934440334 >> 172.21.15.63:0/3934440334 conn(0x7fdca40617a0 msgr2=0x7fdca4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.259+0000 7fdcabc2e640 1 -- 172.21.15.63:0/3934440334 shutdown_connections 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.259+0000 7fdcabc2e640 1 -- 172.21.15.63:0/3934440334 wait complete. 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 10 monclient: build_initial_monmap 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 10 monclient: monmap: 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:46.266 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 1 librados: starting msgr at 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 1 librados: starting objecter 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 1 Processor -- start 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 1 -- start start 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 1 librados: setting wanted keys 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 1 librados: calling monclient init 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.260+0000 7fdcabc2e640 10 monclient: init 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:46.267 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 10 monclient(hunting): picked mon.noname-a con 0x7fdca40b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 10 monclient(hunting): picked mon.noname-b con 0x7fdca40b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 10 monclient(hunting): picked mon.noname-c con 0x7fdca414be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40aabb0 con 0x7fdca40b1730 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40a9900 con 0x7fdca40b4d60 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40a9470 con 0x7fdca414be90 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:46.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.261+0000 7fdcabc2e640 10 monclient(hunting): authenticate will time out at 1889.405925s 2024-03-25T14:28:46.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.273+0000 7fdca99ca640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fdca40b1730 0x7fdca4140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:41388/0 (socket says 172.21.15.63:41388) 2024-03-25T14:28:46.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.273+0000 7fdca99ca640 1 -- 172.21.15.63:0/2591546457 learned_addr learned my addr 172.21.15.63:0/2591546457 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:46.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.274+0000 7fdc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fdca40b1730 2024-03-25T14:28:46.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.274+0000 7fdc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:46.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.278+0000 7fdc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fdca40b4d60 2024-03-25T14:28:46.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.278+0000 7fdc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:46.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.282+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175933917 0 0) 0x7fdca40aabb0 con 0x7fdca40b1730 2024-03-25T14:28:46.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.282+0000 7fdc9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.282+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.282+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.282+0000 7fdc9a7fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T14:28:46.287 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.282+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc88004eb0 con 0x7fdca40b1730 2024-03-25T14:28:46.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.284+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815461441 0 0) 0x7fdca40a9900 con 0x7fdca40b4d60 2024-03-25T14:28:46.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.284+0000 7fdc9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.288 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.284+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.284+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.284+0000 7fdc9a7fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T14:28:46.289 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.284+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca40aabb0 con 0x7fdca40b4d60 2024-03-25T14:28:46.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.286+0000 7fdc9a7fc640 10 client.?.objecter ms_handle_connect 0x7fdca414be90 2024-03-25T14:28:46.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.286+0000 7fdc9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.293+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 279461408 0 0) 0x7fdc88004eb0 con 0x7fdca40b1730 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.293+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca40a9900 con 0x7fdca40b1730 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.293+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdca0001a10 con 0x7fdca40b1730 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.293+0000 7fdc9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.293+0000 7fdc9a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.293+0000 7fdc9a7fc640 10 monclient(hunting): dump: 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.298 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:46.299 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:46.299 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:46.299 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:46.299 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:46.299 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.293+0000 7fdc9a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:46.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.296+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591209662 0 0) 0x7fdca40a9470 con 0x7fdca414be90 2024-03-25T14:28:46.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.296+0000 7fdc9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:28:46.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.296+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:46.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.296+0000 7fdc9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:46.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.296+0000 7fdc9a7fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T14:28:46.301 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.296+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc88004eb0 con 0x7fdca414be90 2024-03-25T14:28:46.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2156140780 0 0) 0x7fdca40a9900 con 0x7fdca40b1730 2024-03-25T14:28:46.305 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 >> v1:172.21.15.63:6790/0 conn(0x7fdca414be90 legacy=0x7fdca4137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 >> v1:172.21.15.153:6789/0 conn(0x7fdca40b4d60 legacy=0x7fdca4136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 1 monclient: found mon.a 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40a9600 con 0x7fdca40b1730 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdc9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 5 monclient: authenticate success, global_id 4626 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 10 monclient: _renew_subs 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:46.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca4137830 con 0x7fdca40b1730 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 10 monclient: _renew_subs 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.300+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4143610 con 0x7fdca40b1730 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.301+0000 7fdcabc2e640 1 librados: init done 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.301+0000 7fdcabc2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.301+0000 7fdcabc2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.301+0000 7fdcabc2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:46.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.301+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca40a9a90 con 0x7fdca40b1730 2024-03-25T14:28:46.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdca0001c70 con 0x7fdca40b1730 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdca0003a60 con 0x7fdca40b1730 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9a7fc640 10 monclient: dump: 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:46.630 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:46.631 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:46.631 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:46.631 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:46.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9bfff640 4 set_mon_vals no callback set 2024-03-25T14:28:46.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.625+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fdca0003c80 con 0x7fdca40b1730 2024-03-25T14:28:46.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.626+0000 7fdca91c9640 10 monclient: get_auth_request con 0x7fdc8806de00 auth_method 0 2024-03-25T14:28:46.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.626+0000 7fdc9a7fc640 10 client.4626.objecter ms_handle_connect 0x7fdc8806de00 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fdca0054860 con 0x7fdca40b1730 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 10 client.4626.objecter ms_dispatch 0x7fdca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdca0050370 con 0x7fdca40b1730 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:46.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.632+0000 7fdc9a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:46.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.888+0000 7fdcabc2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:46.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.888+0000 7fdcabc2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:46.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.888+0000 7fdcabc2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:28:46.893 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.888+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdca4061c30 con 0x7fdca40b1730 2024-03-25T14:28:46.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.913+0000 7fdc9a7fc640 1 -- 172.21.15.63:0/2591546457 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fdca00522a0 con 0x7fdca40b1730 2024-03-25T14:28:46.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.913+0000 7fdc9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:46.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.913+0000 7fdc9a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:46.918 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:46.918 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:46.920 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 10 librados: watch_flush enter 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 10 librados: watch_flush exit 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 10 client.4626.objecter successfully canceled tick 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 >> v1:172.21.15.153:6800/1688255552 conn(0x7fdc8806de00 legacy=0x7fdc880702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 10 monclient: shutdown 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 >> v1:172.21.15.63:6789/0 conn(0x7fdca40b1730 legacy=0x7fdca4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 shutdown_connections 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 >> 172.21.15.63:0/2591546457 conn(0x7fdca40617a0 msgr2=0x7fdca413c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 shutdown_connections 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 1 -- 172.21.15.63:0/2591546457 wait complete. 2024-03-25T14:28:46.921 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:46.915+0000 7fdcabc2e640 1 librados: shutdown 2024-03-25T14:28:46.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:28:52.659 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:31.170 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.718 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:31.231 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.736 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:31.248 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.762 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:31.273 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.773 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:31.232 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.775 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:31.287 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.781 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:31.240 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.813 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:31.325 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:52.819 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:31.278 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:28:53.739 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.833+0000 7f65213c8640 10 monclient: get_monmap_and_config 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.833+0000 7f65213c8640 10 monclient: build_initial_monmap 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.833+0000 7f65213c8640 10 monclient: monmap: 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:53.838 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:28:53.834569+0000 2024-03-25T14:28:53.839 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:28:53.834569+0000 2024-03-25T14:28:53.839 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:53.839 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:53.839 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:53.839 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:28:53.839 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:28:53.839 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:53.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 1 Processor -- start 2024-03-25T14:28:53.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 1 -- start start 2024-03-25T14:28:53.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient: init 2024-03-25T14:28:53.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:53.840 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient(hunting): picked mon.noname-b con 0x7f651c0b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient(hunting): picked mon.noname-c con 0x7f651c0b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient(hunting): picked mon.noname-a con 0x7f651c14be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0af940 con 0x7f651c14be90 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0b0a70 con 0x7f651c0b1730 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0b0d50 con 0x7f651c0b4d60 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:53.841 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.834+0000 7f65213c8640 10 monclient(hunting): authenticate will time out at 1896.979387s 2024-03-25T14:28:53.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.846+0000 7f6513fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f651c0b1730 0x7f651c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:51870/0 (socket says 172.21.15.63:51870) 2024-03-25T14:28:53.851 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.846+0000 7f6513fff640 1 -- 172.21.15.63:0/2799911747 learned_addr learned my addr 172.21.15.63:0/2799911747 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:53.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.866+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289482951 0 0) 0x7f651c0b0a70 con 0x7f651c0b1730 2024-03-25T14:28:53.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.866+0000 7f651a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.866+0000 7f651a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.871 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.866+0000 7f651a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.866+0000 7f651a7fc640 10 monclient(hunting): my global_id is 4633 2024-03-25T14:28:53.872 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.866+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6508003520 con 0x7f651c0b1730 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2441324880 0 0) 0x7f6508003520 con 0x7f651c0b1730 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 >> v1:172.21.15.63:6790/0 conn(0x7f651c0b4d60 legacy=0x7f651c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 >> v1:172.21.15.63:6789/0 conn(0x7f651c14be90 legacy=0x7f651c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 1 monclient: found mon.noname-c 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c0b1030 con 0x7f651c0b1730 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:53.877 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.872+0000 7f65213c8640 5 monclient: authenticate success, global_id 4633 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.872+0000 7f65213c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.871+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6504001690 con 0x7f651c0b1730 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.872+0000 7f651a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.872+0000 7f651a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.872+0000 7f651a7fc640 10 monclient: dump: 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:53.878 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:53.879 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:53.879 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:53.879 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.872+0000 7f65213c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65040018b0 con 0x7f651c0b1730 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f651a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f651a7fc640 1 -- 172.21.15.63:0/2799911747 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6504001c40 con 0x7f651c0b1730 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f651a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f651a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f651a7fc640 10 monclient: dump: 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.898 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f65213c8640 10 monclient: get_monmap_and_config success 2024-03-25T14:28:53.899 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.893+0000 7f65213c8640 4 set_mon_vals no callback set 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.894+0000 7f65213c8640 10 monclient: shutdown 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.894+0000 7f65213c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.894+0000 7f65213c8640 1 -- 172.21.15.63:0/2799911747 >> v1:172.21.15.153:6789/0 conn(0x7f651c0b1730 legacy=0x7f651c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.894+0000 7f65213c8640 1 -- 172.21.15.63:0/2799911747 shutdown_connections 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.894+0000 7f65213c8640 1 -- 172.21.15.63:0/2799911747 >> 172.21.15.63:0/2799911747 conn(0x7f651c0617a0 msgr2=0x7f651c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.894+0000 7f65213c8640 1 -- 172.21.15.63:0/2799911747 shutdown_connections 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.894+0000 7f65213c8640 1 -- 172.21.15.63:0/2799911747 wait complete. 2024-03-25T14:28:53.901 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 10 monclient: build_initial_monmap 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 10 monclient: monmap: 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 1 librados: starting msgr at 2024-03-25T14:28:53.902 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 1 librados: starting objecter 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 1 Processor -- start 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 1 -- start start 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 1 librados: setting wanted keys 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 1 librados: calling monclient init 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.895+0000 7f65213c8640 10 monclient: init 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 10 monclient: _reopen_session rank -1 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 10 monclient(hunting): picked mon.noname-c con 0x7f651c0b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 10 monclient(hunting): picked mon.noname-a con 0x7f651c0b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 10 monclient(hunting): picked mon.noname-b con 0x7f651c14be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c1d83d0 con 0x7f651c0b4d60 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c1d77a0 con 0x7f651c14be90 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c138270 con 0x7f651c0b1730 2024-03-25T14:28:53.903 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 10 monclient(hunting): _renew_subs 2024-03-25T14:28:53.904 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.896+0000 7f65213c8640 10 monclient(hunting): authenticate will time out at 1897.041030s 2024-03-25T14:28:53.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.909+0000 7f651affd640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f651c0b1730 0x7f651c1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:45638/0 (socket says 172.21.15.63:45638) 2024-03-25T14:28:53.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.909+0000 7f651affd640 1 -- 172.21.15.63:0/3002042166 learned_addr learned my addr 172.21.15.63:0/3002042166 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:28:53.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.911+0000 7f65137fe640 10 client.?.objecter ms_handle_connect 0x7f651c14be90 2024-03-25T14:28:53.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.911+0000 7f65137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:53.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.911+0000 7f65137fe640 10 client.?.objecter ms_handle_connect 0x7f651c0b1730 2024-03-25T14:28:53.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.911+0000 7f65137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:53.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.911+0000 7f65137fe640 10 client.?.objecter ms_handle_connect 0x7f651c0b4d60 2024-03-25T14:28:53.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.911+0000 7f65137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:28:53.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.921+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280232599 0 0) 0x7f651c1d83d0 con 0x7f651c0b4d60 2024-03-25T14:28:53.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.921+0000 7f65137fe640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.926 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.921+0000 7f65137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.921+0000 7f65137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.921+0000 7f65137fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T14:28:53.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.921+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64fc004eb0 con 0x7f651c0b4d60 2024-03-25T14:28:53.935 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.931+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160354059 0 0) 0x7f651c1d77a0 con 0x7f651c14be90 2024-03-25T14:28:53.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.931+0000 7f65137fe640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.931+0000 7f65137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.931+0000 7f65137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.931+0000 7f65137fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T14:28:53.936 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.931+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c1d83d0 con 0x7f651c14be90 2024-03-25T14:28:53.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.939+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762674115 0 0) 0x7f651c138270 con 0x7f651c0b1730 2024-03-25T14:28:53.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.939+0000 7f65137fe640 10 monclient(hunting): _negotiate 2024-03-25T14:28:53.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.939+0000 7f65137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:28:53.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.939+0000 7f65137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:28:53.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.939+0000 7f65137fe640 10 monclient(hunting): my global_id is 4610 2024-03-25T14:28:53.944 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.939+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c1d77a0 con 0x7f651c0b1730 2024-03-25T14:28:53.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.952+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2543062368 0 0) 0x7f64fc004eb0 con 0x7f651c0b4d60 2024-03-25T14:28:53.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.952+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f651c138270 con 0x7f651c0b4d60 2024-03-25T14:28:53.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.952+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6504001610 con 0x7f651c0b4d60 2024-03-25T14:28:53.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.952+0000 7f65137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.952+0000 7f65137fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:28:53.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.952+0000 7f65137fe640 10 monclient(hunting): dump: 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:53.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.952+0000 7f65137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:28:53.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.964+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2062447909 0 0) 0x7f651c1d77a0 con 0x7f651c0b1730 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.965+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64fc004eb0 con 0x7f651c0b1730 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.965+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f650c0018f0 con 0x7f651c0b1730 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.965+0000 7f65137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.965+0000 7f65137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.965+0000 7f65137fe640 10 monclient(hunting): dump: 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:53.970 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:53.971 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:53.971 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:53.971 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:53.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.971+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687923997 0 0) 0x7f651c1d83d0 con 0x7f651c14be90 2024-03-25T14:28:53.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.971+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f651c1d77a0 con 0x7f651c14be90 2024-03-25T14:28:53.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.971+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6514001b10 con 0x7f651c14be90 2024-03-25T14:28:53.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.971+0000 7f65137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:53.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.971+0000 7f65137fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:53.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.971+0000 7f65137fe640 10 monclient(hunting): dump: 2024-03-25T14:28:53.976 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:53.977 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:53.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 820032473 0 0) 0x7f651c1d77a0 con 0x7f651c14be90 2024-03-25T14:28:53.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 >> v1:172.21.15.63:6790/0 conn(0x7f651c0b1730 legacy=0x7f651c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 >> v1:172.21.15.63:6789/0 conn(0x7f651c0b4d60 legacy=0x7f651c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 10 monclient: _finish_hunting 0 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 1 monclient: found mon.b 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c140d00 con 0x7f651c14be90 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 10 monclient: _finish_auth 0 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 10 monclient: _check_auth_tickets 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 5 monclient: authenticate success, global_id 4636 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 10 monclient: _renew_subs 2024-03-25T14:28:53.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c140ef0 con 0x7f651c14be90 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 10 monclient: _renew_subs 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c138c60 con 0x7f651c14be90 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.976+0000 7f65213c8640 1 librados: init done 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.977+0000 7f65213c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:53.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.977+0000 7f65213c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:53.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.977+0000 7f65213c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:53.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:53.977+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c0afbd0 con 0x7f651c14be90 2024-03-25T14:28:54.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.263+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6514001fe0 con 0x7f651c14be90 2024-03-25T14:28:54.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.263+0000 7f65137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:28:54.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.263+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f6514003970 con 0x7f651c14be90 2024-03-25T14:28:54.268 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.263+0000 7f65137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.263+0000 7f65137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.263+0000 7f65137fe640 10 monclient: dump: 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:28:54.269 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.263+0000 7f6519ffb640 4 set_mon_vals no callback set 2024-03-25T14:28:54.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.264+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f6514003b90 con 0x7f651c14be90 2024-03-25T14:28:54.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.265+0000 7f651affd640 10 monclient: get_auth_request con 0x7f64fc06e110 auth_method 0 2024-03-25T14:28:54.270 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.265+0000 7f65137fe640 10 client.4636.objecter ms_handle_connect 0x7f64fc06e110 2024-03-25T14:28:54.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.268+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f6514053580 con 0x7f651c14be90 2024-03-25T14:28:54.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.268+0000 7f65137fe640 10 client.4636.objecter ms_dispatch 0x7f651c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:28:54.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.268+0000 7f65137fe640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:28:54.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.268+0000 7f65137fe640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:28:54.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.268+0000 7f65137fe640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T14:28:54.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.268+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f651404f090 con 0x7f651c14be90 2024-03-25T14:28:54.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.269+0000 7f65137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:28:54.274 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.269+0000 7f65137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:28:54.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.525+0000 7f65213c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:54.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.525+0000 7f65213c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:54.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.525+0000 7f65213c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:28:54.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.525+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f651c0627b0 con 0x7f651c14be90 2024-03-25T14:28:54.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.557+0000 7f65137fe640 1 -- 172.21.15.63:0/3002042166 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f6514050fc0 con 0x7f651c14be90 2024-03-25T14:28:54.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.557+0000 7f65137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:28:54.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.557+0000 7f65137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:28:54.562 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:28:54.563 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.559+0000 7f65213c8640 10 librados: watch_flush enter 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 10 librados: watch_flush exit 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 10 client.4636.objecter successfully canceled tick 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 >> v1:172.21.15.153:6800/1688255552 conn(0x7f64fc06e110 legacy=0x7f64fc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 10 monclient: shutdown 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 >> v1:172.21.15.153:6789/0 conn(0x7f651c14be90 legacy=0x7f651c142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:28:54.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 shutdown_connections 2024-03-25T14:28:54.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 >> 172.21.15.63:0/3002042166 conn(0x7f651c0617a0 msgr2=0x7f651c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:28:54.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 shutdown_connections 2024-03-25T14:28:54.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 1 -- 172.21.15.63:0/3002042166 wait complete. 2024-03-25T14:28:54.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:28:54.560+0000 7f65213c8640 1 librados: shutdown 2024-03-25T14:28:54.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:01.385 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:01.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.486+0000 7f5d721e2640 10 monclient: get_monmap_and_config 2024-03-25T14:29:01.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.486+0000 7f5d721e2640 10 monclient: build_initial_monmap 2024-03-25T14:29:01.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.486+0000 7f5d721e2640 10 monclient: monmap: 2024-03-25T14:29:01.491 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:01.487871+0000 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:01.487871+0000 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:01.492 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:01.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.487+0000 7f5d721e2640 1 Processor -- start 2024-03-25T14:29:01.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.487+0000 7f5d721e2640 1 -- start start 2024-03-25T14:29:01.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.487+0000 7f5d721e2640 10 monclient: init 2024-03-25T14:29:01.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:01.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:29:01.493 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5d6c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5d6c13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5d6c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c142520 con 0x7f5d6c1430c0 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0b28a0 con 0x7f5d6c13f710 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c0b2a30 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.488+0000 7f5d721e2640 10 monclient(hunting): authenticate will time out at 1904.633062s 2024-03-25T14:29:01.494 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.490+0000 7f5d6affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5d6c13f710 0x7f5d6c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:54596/0 (socket says 172.21.15.63:54596) 2024-03-25T14:29:01.495 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.490+0000 7f5d6affd640 1 -- 172.21.15.63:0/522359949 learned_addr learned my addr 172.21.15.63:0/522359949 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:01.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.557+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578969933 0 0) 0x7f5d6c142520 con 0x7f5d6c1430c0 2024-03-25T14:29:01.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.557+0000 7f5d6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.557+0000 7f5d6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.557+0000 7f5d6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.557+0000 7f5d6a7fc640 10 monclient(hunting): my global_id is 4635 2024-03-25T14:29:01.563 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.557+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d54003520 con 0x7f5d6c1430c0 2024-03-25T14:29:01.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115922830 0 0) 0x7f5d6c0b28a0 con 0x7f5d6c13f710 2024-03-25T14:29:01.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient(hunting): my global_id is 4639 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c142520 con 0x7f5d6c13f710 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1073693370 0 0) 0x7f5d54003520 con 0x7f5d6c1430c0 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 >> v1:172.21.15.63:6790/0 conn(0x7f5d6c0a98f0 legacy=0x7f5d6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 >> v1:172.21.15.153:6789/0 conn(0x7f5d6c13f710 legacy=0x7f5d6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 monclient: found mon.noname-a 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c0b44c0 con 0x7f5d6c1430c0 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient: _finish_auth 0 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:01.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5d64001a10 con 0x7f5d6c1430c0 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d6a7fc640 10 monclient: dump: 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:01.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d721e2640 5 monclient: authenticate success, global_id 4635 2024-03-25T14:29:01.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.571+0000 7f5d721e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:01.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d64001b70 con 0x7f5d6c1430c0 2024-03-25T14:29:01.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:01.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d6a7fc640 1 -- 172.21.15.63:0/522359949 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5d64001f80 con 0x7f5d6c1430c0 2024-03-25T14:29:01.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:01.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d6a7fc640 10 monclient: dump: 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d721e2640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:01.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.596+0000 7f5d721e2640 4 set_mon_vals no callback set 2024-03-25T14:29:01.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.597+0000 7f5d721e2640 10 monclient: shutdown 2024-03-25T14:29:01.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.597+0000 7f5d721e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:01.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.597+0000 7f5d721e2640 1 -- 172.21.15.63:0/522359949 >> v1:172.21.15.63:6789/0 conn(0x7f5d6c1430c0 legacy=0x7f5d6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.597+0000 7f5d721e2640 1 -- 172.21.15.63:0/522359949 shutdown_connections 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.597+0000 7f5d721e2640 1 -- 172.21.15.63:0/522359949 >> 172.21.15.63:0/522359949 conn(0x7f5d6c0b7ea0 msgr2=0x7f5d6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.597+0000 7f5d721e2640 1 -- 172.21.15.63:0/522359949 shutdown_connections 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.597+0000 7f5d721e2640 1 -- 172.21.15.63:0/522359949 wait complete. 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.598+0000 7f5d721e2640 10 monclient: build_initial_monmap 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.598+0000 7f5d721e2640 10 monclient: monmap: 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:01.606 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.598+0000 7f5d721e2640 1 librados: starting msgr at 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.598+0000 7f5d721e2640 1 librados: starting objecter 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.598+0000 7f5d721e2640 1 Processor -- start 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 1 -- start start 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 1 librados: setting wanted keys 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 1 librados: calling monclient init 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient: init 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5d6c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:01.607 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient(hunting): picked mon.noname-b con 0x7f5d6c13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:01.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5d6c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:01.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c139830 con 0x7f5d6c1430c0 2024-03-25T14:29:01.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c1db7d0 con 0x7f5d6c13f710 2024-03-25T14:29:01.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6c1db410 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:01.608 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.599+0000 7f5d721e2640 10 monclient(hunting): authenticate will time out at 1904.744260s 2024-03-25T14:29:01.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.614+0000 7f5d6affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5d6c13f710 0x7f5d6c0b0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:54640/0 (socket says 172.21.15.63:54640) 2024-03-25T14:29:01.618 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.614+0000 7f5d6affd640 1 -- 172.21.15.63:0/2264984257 learned_addr learned my addr 172.21.15.63:0/2264984257 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:01.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.614+0000 7f5d53fff640 10 client.?.objecter ms_handle_connect 0x7f5d6c13f710 2024-03-25T14:29:01.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.614+0000 7f5d53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:01.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.621+0000 7f5d53fff640 10 client.?.objecter ms_handle_connect 0x7f5d6c0a98f0 2024-03-25T14:29:01.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.621+0000 7f5d53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:01.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.624+0000 7f5d53fff640 10 client.?.objecter ms_handle_connect 0x7f5d6c1430c0 2024-03-25T14:29:01.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.624+0000 7f5d53fff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:01.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.634+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653618011 0 0) 0x7f5d6c1db410 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.634+0000 7f5d53fff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.634+0000 7f5d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.634+0000 7f5d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.634+0000 7f5d53fff640 10 monclient(hunting): my global_id is 4616 2024-03-25T14:29:01.639 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.634+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d58004eb0 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.636+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536064080 0 0) 0x7f5d6c1db7d0 con 0x7f5d6c13f710 2024-03-25T14:29:01.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.636+0000 7f5d53fff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.636+0000 7f5d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.640 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.636+0000 7f5d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.636+0000 7f5d53fff640 10 monclient(hunting): my global_id is 4642 2024-03-25T14:29:01.641 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.636+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c1db410 con 0x7f5d6c13f710 2024-03-25T14:29:01.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.647+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261560967 0 0) 0x7f5d6c139830 con 0x7f5d6c1430c0 2024-03-25T14:29:01.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.647+0000 7f5d53fff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:01.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.647+0000 7f5d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:01.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.647+0000 7f5d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:01.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.647+0000 7f5d53fff640 10 monclient(hunting): my global_id is 4638 2024-03-25T14:29:01.652 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.647+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c1db7d0 con 0x7f5d6c1430c0 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.653+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189057953 0 0) 0x7f5d58004eb0 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.653+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d6c139830 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.653+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5d5c0018c0 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.653+0000 7f5d53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.653+0000 7f5d53fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.653+0000 7f5d53fff640 10 monclient(hunting): dump: 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.658 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:01.659 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:01.659 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:01.659 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:01.659 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:01.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.653+0000 7f5d53fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:01.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.660+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3154197432 0 0) 0x7f5d6c1db7d0 con 0x7f5d6c1430c0 2024-03-25T14:29:01.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.660+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d58004eb0 con 0x7f5d6c1430c0 2024-03-25T14:29:01.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.660+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5d64001e80 con 0x7f5d6c1430c0 2024-03-25T14:29:01.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.660+0000 7f5d53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.660+0000 7f5d53fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.660+0000 7f5d53fff640 10 monclient(hunting): dump: 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:01.666 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:01.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1405461408 0 0) 0x7f5d6c139830 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 >> v1:172.21.15.153:6789/0 conn(0x7f5d6c13f710 legacy=0x7f5d6c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 >> v1:172.21.15.63:6789/0 conn(0x7f5d6c1430c0 legacy=0x7f5d6c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 10 monclient: _finish_hunting 0 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 1 monclient: found mon.c 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c1db5a0 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 10 monclient: _finish_auth 0 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 10 monclient: _check_auth_tickets 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 5 monclient: authenticate success, global_id 4616 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 10 monclient: _renew_subs 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:01.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c0b1480 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:01.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 10 monclient: _renew_subs 2024-03-25T14:29:01.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:01.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c0b0b10 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.662+0000 7f5d721e2640 1 librados: init done 2024-03-25T14:29:01.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.663+0000 7f5d721e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:01.669 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.663+0000 7f5d721e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:01.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.663+0000 7f5d721e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:01.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.663+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c0627b0 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.977+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d5c0035f0 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.977+0000 7f5d53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:01.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.977+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5d5c003a00 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.977+0000 7f5d53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:01.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.977+0000 7f5d53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.977+0000 7f5d53fff640 10 monclient: dump: 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:01.983 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:01.984 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:01.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.977+0000 7f5d69ffb640 4 set_mon_vals no callback set 2024-03-25T14:29:01.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.978+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f5d5c003c20 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.979+0000 7f5d6affd640 10 monclient: get_auth_request con 0x7f5d5806df70 auth_method 0 2024-03-25T14:29:01.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.979+0000 7f5d53fff640 10 client.4616.objecter ms_handle_connect 0x7f5d5806df70 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f5d5c053600 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 10 client.4616.objecter ms_dispatch 0x7f5d6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d5c04f110 con 0x7f5d6c0a98f0 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:01.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:01.987+0000 7f5d53fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:02.239 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.234+0000 7f5d721e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:02.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.234+0000 7f5d721e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:02.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.234+0000 7f5d721e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:02.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.234+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d6c1db960 con 0x7f5d6c0a98f0 2024-03-25T14:29:02.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.256+0000 7f5d53fff640 1 -- 172.21.15.63:0/2264984257 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f5d5c051040 con 0x7f5d6c0a98f0 2024-03-25T14:29:02.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.256+0000 7f5d53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:02.261 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.256+0000 7f5d53fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:02.261 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:02.261 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 10 librados: watch_flush enter 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 10 librados: watch_flush exit 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 10 client.4616.objecter successfully canceled tick 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 >> v1:172.21.15.153:6800/1688255552 conn(0x7f5d5806df70 legacy=0x7f5d58070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 10 monclient: shutdown 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 >> v1:172.21.15.63:6790/0 conn(0x7f5d6c0a98f0 legacy=0x7f5d6c1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:02.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 shutdown_connections 2024-03-25T14:29:02.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 >> 172.21.15.63:0/2264984257 conn(0x7f5d6c0b7ea0 msgr2=0x7f5d6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:02.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 shutdown_connections 2024-03-25T14:29:02.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 1 -- 172.21.15.63:0/2264984257 wait complete. 2024-03-25T14:29:02.264 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:02.258+0000 7f5d721e2640 1 librados: shutdown 2024-03-25T14:29:02.275 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:09.081 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:09.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.189+0000 7f67bbdc6640 10 monclient: get_monmap_and_config 2024-03-25T14:29:09.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.189+0000 7f67bbdc6640 10 monclient: build_initial_monmap 2024-03-25T14:29:09.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.189+0000 7f67bbdc6640 10 monclient: monmap: 2024-03-25T14:29:09.194 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:09.194 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:09.194 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:09.190567+0000 2024-03-25T14:29:09.195 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:09.190567+0000 2024-03-25T14:29:09.195 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:09.195 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:09.195 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:09.195 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:09.195 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:09.195 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:09.196 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.190+0000 7f67bbdc6640 1 Processor -- start 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.190+0000 7f67bbdc6640 1 -- start start 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.190+0000 7f67bbdc6640 10 monclient: init 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f67b40a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f67b413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f67b41430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b4142520 con 0x7f67b413f710 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b40b28a0 con 0x7f67b40a98f0 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b40b2a30 con 0x7f67b41430c0 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:09.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.191+0000 7f67bbdc6640 10 monclient(hunting): authenticate will time out at 1912.336447s 2024-03-25T14:29:09.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.201+0000 7f67ba363640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f67b41430c0 0x7f67b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:48214/0 (socket says 172.21.15.63:48214) 2024-03-25T14:29:09.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.201+0000 7f67ba363640 1 -- 172.21.15.63:0/162555919 learned_addr learned my addr 172.21.15.63:0/162555919 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:09.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.211+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361130796 0 0) 0x7f67b40b2a30 con 0x7f67b41430c0 2024-03-25T14:29:09.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.211+0000 7f67b8b60640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.211+0000 7f67b8b60640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.211+0000 7f67b8b60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.211+0000 7f67b8b60640 10 monclient(hunting): my global_id is 4619 2024-03-25T14:29:09.217 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.211+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a0003520 con 0x7f67b41430c0 2024-03-25T14:29:09.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.229+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292853110 0 0) 0x7f67b4142520 con 0x7f67b413f710 2024-03-25T14:29:09.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.229+0000 7f67b8b60640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.229+0000 7f67b8b60640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.229+0000 7f67b8b60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.229+0000 7f67b8b60640 10 monclient(hunting): my global_id is 4641 2024-03-25T14:29:09.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.229+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b40b2a30 con 0x7f67b413f710 2024-03-25T14:29:09.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.238+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783758759 0 0) 0x7f67b40b28a0 con 0x7f67b40a98f0 2024-03-25T14:29:09.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.238+0000 7f67b8b60640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.238+0000 7f67b8b60640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.238+0000 7f67b8b60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.238+0000 7f67b8b60640 10 monclient(hunting): my global_id is 4645 2024-03-25T14:29:09.243 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.238+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b4142520 con 0x7f67b40a98f0 2024-03-25T14:29:09.246 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112079037 0 0) 0x7f67b40b2a30 con 0x7f67b413f710 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 >> v1:172.21.15.63:6790/0 conn(0x7f67b41430c0 legacy=0x7f67b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 >> v1:172.21.15.153:6789/0 conn(0x7f67b40a98f0 legacy=0x7f67b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 10 monclient: _finish_hunting 0 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 1 monclient: found mon.noname-a 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b40b44c0 con 0x7f67b413f710 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 10 monclient: _finish_auth 0 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 10 monclient: _check_auth_tickets 2024-03-25T14:29:09.247 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f67a4001670 con 0x7f67b413f710 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67b8b60640 10 monclient: dump: 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:09.248 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:09.249 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:09.249 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:09.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67bbdc6640 5 monclient: authenticate success, global_id 4641 2024-03-25T14:29:09.249 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.241+0000 7f67bbdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67a40017d0 con 0x7f67b413f710 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67b8b60640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67b8b60640 1 -- 172.21.15.63:0/162555919 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f67a4001be0 con 0x7f67b413f710 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67bbdc6640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67bbdc6640 4 set_mon_vals no callback set 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67b8b60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67b8b60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.248+0000 7f67b8b60640 10 monclient: dump: 2024-03-25T14:29:09.253 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:09.254 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:09.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.249+0000 7f67bbdc6640 10 monclient: shutdown 2024-03-25T14:29:09.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.249+0000 7f67bbdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:09.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.249+0000 7f67bbdc6640 1 -- 172.21.15.63:0/162555919 >> v1:172.21.15.63:6789/0 conn(0x7f67b413f710 legacy=0x7f67b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.249+0000 7f67bbdc6640 1 -- 172.21.15.63:0/162555919 shutdown_connections 2024-03-25T14:29:09.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.249+0000 7f67bbdc6640 1 -- 172.21.15.63:0/162555919 >> 172.21.15.63:0/162555919 conn(0x7f67b40b7ea0 msgr2=0x7f67b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:09.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.249+0000 7f67bbdc6640 1 -- 172.21.15.63:0/162555919 shutdown_connections 2024-03-25T14:29:09.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.249+0000 7f67bbdc6640 1 -- 172.21.15.63:0/162555919 wait complete. 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 10 monclient: build_initial_monmap 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 10 monclient: monmap: 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:09.258 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 1 librados: starting msgr at 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 1 librados: starting objecter 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 1 Processor -- start 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 1 -- start start 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 1 librados: setting wanted keys 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 1 librados: calling monclient init 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.250+0000 7f67bbdc6640 10 monclient: init 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:09.259 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f67b40a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f67b413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f67b41430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b41dad00 con 0x7f67b41430c0 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b4139a10 con 0x7f67b40a98f0 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b41db8a0 con 0x7f67b413f710 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.251+0000 7f67bbdc6640 10 monclient(hunting): authenticate will time out at 1912.396097s 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.252+0000 7f67b9b62640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f67b40a98f0 0x7f67b4137b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:59344/0 (socket says 172.21.15.63:59344) 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.252+0000 7f67b9b62640 1 -- 172.21.15.63:0/2087848490 learned_addr learned my addr 172.21.15.63:0/2087848490 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.252+0000 7f67aa7fc640 10 client.?.objecter ms_handle_connect 0x7f67b40a98f0 2024-03-25T14:29:09.260 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.252+0000 7f67aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:09.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.258+0000 7f67aa7fc640 10 client.?.objecter ms_handle_connect 0x7f67b413f710 2024-03-25T14:29:09.263 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.258+0000 7f67aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:09.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.313+0000 7f67aa7fc640 10 client.?.objecter ms_handle_connect 0x7f67b41430c0 2024-03-25T14:29:09.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.313+0000 7f67aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:09.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.339+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026740705 0 0) 0x7f67b41dad00 con 0x7f67b41430c0 2024-03-25T14:29:09.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.339+0000 7f67aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.339+0000 7f67aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.339+0000 7f67aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.339+0000 7f67aa7fc640 10 monclient(hunting): my global_id is 4644 2024-03-25T14:29:09.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.339+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6798004eb0 con 0x7f67b41430c0 2024-03-25T14:29:09.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.361+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 149736492 0 0) 0x7f6798004eb0 con 0x7f67b41430c0 2024-03-25T14:29:09.366 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.361+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67b41dad00 con 0x7f67b41430c0 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.362+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f67ac001af0 con 0x7f67b41430c0 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.362+0000 7f67aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.362+0000 7f67aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.362+0000 7f67aa7fc640 10 monclient(hunting): dump: 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.367 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.368 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.368 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:09.368 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:09.368 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:09.368 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:09.368 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:09.368 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.362+0000 7f67aa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:09.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.366+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182145692 0 0) 0x7f67b4139a10 con 0x7f67b40a98f0 2024-03-25T14:29:09.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.366+0000 7f67aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.366+0000 7f67aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.371 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.366+0000 7f67aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.366+0000 7f67aa7fc640 10 monclient(hunting): my global_id is 4648 2024-03-25T14:29:09.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.366+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6798004eb0 con 0x7f67b40a98f0 2024-03-25T14:29:09.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.380+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36617758 0 0) 0x7f67b41db8a0 con 0x7f67b413f710 2024-03-25T14:29:09.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.380+0000 7f67aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:09.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.380+0000 7f67aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:09.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.380+0000 7f67aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:09.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.380+0000 7f67aa7fc640 10 monclient(hunting): my global_id is 4622 2024-03-25T14:29:09.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.380+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b4139a10 con 0x7f67b413f710 2024-03-25T14:29:09.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2931373426 0 0) 0x7f67b41dad00 con 0x7f67b41430c0 2024-03-25T14:29:09.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 >> v1:172.21.15.63:6790/0 conn(0x7f67b413f710 legacy=0x7f67b41db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 >> v1:172.21.15.153:6789/0 conn(0x7f67b40a98f0 legacy=0x7f67b4137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T14:29:09.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 1 monclient: found mon.a 2024-03-25T14:29:09.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b4135080 con 0x7f67b41430c0 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 10 monclient: _finish_auth 0 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 5 monclient: authenticate success, global_id 4644 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 10 monclient: _renew_subs 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b4135270 con 0x7f67b41430c0 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 10 client.4644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 10 monclient: _renew_subs 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b4138b00 con 0x7f67b41430c0 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.381+0000 7f67bbdc6640 1 librados: init done 2024-03-25T14:29:09.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.382+0000 7f67bbdc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:09.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.382+0000 7f67bbdc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:09.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.382+0000 7f67bbdc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:29:09.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.382+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b41daa60 con 0x7f67b41430c0 2024-03-25T14:29:09.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.703+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67ac001f60 con 0x7f67b41430c0 2024-03-25T14:29:09.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.703+0000 7f67aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:09.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.703+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f67ac004d00 con 0x7f67b41430c0 2024-03-25T14:29:09.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.703+0000 7f67aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:09.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.703+0000 7f67aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.703+0000 7f67aa7fc640 10 monclient: dump: 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:09.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.703+0000 7f67abfff640 4 set_mon_vals no callback set 2024-03-25T14:29:09.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.704+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f67ac004f20 con 0x7f67b41430c0 2024-03-25T14:29:09.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.705+0000 7f67b9b62640 10 monclient: get_auth_request con 0x7f679806de80 auth_method 0 2024-03-25T14:29:09.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.705+0000 7f67aa7fc640 10 client.4644.objecter ms_handle_connect 0x7f679806de80 2024-03-25T14:29:09.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f67ac054a10 con 0x7f67b41430c0 2024-03-25T14:29:09.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 10 client.4644.objecter ms_dispatch 0x7f67b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:09.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 3 client.4644.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:09.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 3 client.4644.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:09.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 20 client.4644.objecter dump_active .. 0 homeless 2024-03-25T14:29:09.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67ac050520 con 0x7f67b41430c0 2024-03-25T14:29:09.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:09.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.710+0000 7f67aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:09.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.963+0000 7f67bbdc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:09.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.963+0000 7f67bbdc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:09.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.963+0000 7f67bbdc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:29:09.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.963+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67b40627b0 con 0x7f67b41430c0 2024-03-25T14:29:09.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.989+0000 7f67aa7fc640 1 -- 172.21.15.63:0/2087848490 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f67ac052450 con 0x7f67b41430c0 2024-03-25T14:29:09.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.989+0000 7f67aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:09.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.989+0000 7f67aa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:09.994 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:09.994 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 10 librados: watch_flush enter 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 10 librados: watch_flush exit 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 20 client.4644.objecter shutdown clearing up homeless session... 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 10 client.4644.objecter successfully canceled tick 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 >> v1:172.21.15.153:6800/1688255552 conn(0x7f679806de80 legacy=0x7f6798070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 10 monclient: shutdown 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.991+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 >> v1:172.21.15.63:6789/0 conn(0x7f67b41430c0 legacy=0x7f67b41e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.992+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 shutdown_connections 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.992+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 >> 172.21.15.63:0/2087848490 conn(0x7f67b40b7ea0 msgr2=0x7f67b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.992+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 shutdown_connections 2024-03-25T14:29:09.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.992+0000 7f67bbdc6640 1 -- 172.21.15.63:0/2087848490 wait complete. 2024-03-25T14:29:09.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:09.992+0000 7f67bbdc6640 1 librados: shutdown 2024-03-25T14:29:10.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:12.740 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:51.252 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.786 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:51.245 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.796 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:51.255 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.797 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:51.308 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.797 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:51.309 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.831 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:09:51.343 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.831 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:51.291 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.858 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:51.370 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:12.897 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:09:51.408 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:16.815 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.926+0000 7f0f9c415640 10 monclient: get_monmap_and_config 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.926+0000 7f0f9c415640 10 monclient: build_initial_monmap 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.926+0000 7f0f9c415640 10 monclient: monmap: 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:16.927242+0000 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:16.927242+0000 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:16.931 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:16.932 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:16.932 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:16.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.926+0000 7f0f9c415640 1 Processor -- start 2024-03-25T14:29:16.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.927+0000 7f0f9c415640 1 -- start start 2024-03-25T14:29:16.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.927+0000 7f0f9c415640 10 monclient: init 2024-03-25T14:29:16.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:16.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 10 monclient(hunting): picked mon.noname-c con 0x7f0f940a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9413f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 10 monclient(hunting): picked mon.noname-b con 0x7f0f941430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94142520 con 0x7f0f9413f710 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940b28a0 con 0x7f0f940a98f0 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940b2a30 con 0x7f0f941430c0 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:16.934 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.928+0000 7f0f9c415640 10 monclient(hunting): authenticate will time out at 1920.073036s 2024-03-25T14:29:16.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.935+0000 7f0f9a9b2640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0f941430c0 0x7f0f9414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:52638/0 (socket says 172.21.15.63:52638) 2024-03-25T14:29:16.940 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.935+0000 7f0f9a9b2640 1 -- 172.21.15.63:0/105585082 learned_addr learned my addr 172.21.15.63:0/105585082 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:16.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.954+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442922668 0 0) 0x7f0f940b28a0 con 0x7f0f940a98f0 2024-03-25T14:29:16.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.954+0000 7f0f991af640 10 monclient(hunting): _negotiate 2024-03-25T14:29:16.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.954+0000 7f0f991af640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:16.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.954+0000 7f0f991af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:16.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.954+0000 7f0f991af640 10 monclient(hunting): my global_id is 4651 2024-03-25T14:29:16.959 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.954+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80003520 con 0x7f0f940a98f0 2024-03-25T14:29:16.965 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.960+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832093394 0 0) 0x7f0f940b2a30 con 0x7f0f941430c0 2024-03-25T14:29:16.965 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.960+0000 7f0f991af640 10 monclient(hunting): _negotiate 2024-03-25T14:29:16.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.960+0000 7f0f991af640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:16.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.960+0000 7f0f991af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:16.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.960+0000 7f0f991af640 10 monclient(hunting): my global_id is 4625 2024-03-25T14:29:16.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.960+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f940b28a0 con 0x7f0f941430c0 2024-03-25T14:29:16.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.962+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843297469 0 0) 0x7f0f94142520 con 0x7f0f9413f710 2024-03-25T14:29:16.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.962+0000 7f0f991af640 10 monclient(hunting): _negotiate 2024-03-25T14:29:16.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.962+0000 7f0f991af640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:16.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.962+0000 7f0f991af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:16.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.962+0000 7f0f991af640 10 monclient(hunting): my global_id is 4647 2024-03-25T14:29:16.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.962+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f940b2a30 con 0x7f0f9413f710 2024-03-25T14:29:16.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155998407 0 0) 0x7f0f80003520 con 0x7f0f940a98f0 2024-03-25T14:29:16.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 >> v1:172.21.15.63:6790/0 conn(0x7f0f941430c0 legacy=0x7f0f9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:16.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 >> v1:172.21.15.63:6789/0 conn(0x7f0f9413f710 legacy=0x7f0f941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:16.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 10 monclient: _finish_hunting 0 2024-03-25T14:29:16.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 1 monclient: found mon.noname-c 2024-03-25T14:29:16.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f940b44c0 con 0x7f0f940a98f0 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 10 monclient: _finish_auth 0 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 10 monclient: _check_auth_tickets 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f90001690 con 0x7f0f940a98f0 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f991af640 10 monclient: dump: 2024-03-25T14:29:16.971 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f9c415640 5 monclient: authenticate success, global_id 4651 2024-03-25T14:29:16.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.965+0000 7f0f9c415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:16.979 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f900017f0 con 0x7f0f940a98f0 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f991af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f991af640 1 -- 172.21.15.63:0/105585082 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f90001c00 con 0x7f0f940a98f0 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f991af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f991af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f991af640 10 monclient: dump: 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:16.980 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:16.981 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:16.981 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:16.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f9c415640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:16.981 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.975+0000 7f0f9c415640 4 set_mon_vals no callback set 2024-03-25T14:29:16.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.976+0000 7f0f9c415640 10 monclient: shutdown 2024-03-25T14:29:16.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.976+0000 7f0f9c415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:16.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.976+0000 7f0f9c415640 1 -- 172.21.15.63:0/105585082 >> v1:172.21.15.153:6789/0 conn(0x7f0f940a98f0 legacy=0x7f0f940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:16.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.976+0000 7f0f9c415640 1 -- 172.21.15.63:0/105585082 shutdown_connections 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.976+0000 7f0f9c415640 1 -- 172.21.15.63:0/105585082 >> 172.21.15.63:0/105585082 conn(0x7f0f940b7ea0 msgr2=0x7f0f940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.976+0000 7f0f9c415640 1 -- 172.21.15.63:0/105585082 shutdown_connections 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.976+0000 7f0f9c415640 1 -- 172.21.15.63:0/105585082 wait complete. 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 10 monclient: build_initial_monmap 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 10 monclient: monmap: 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:16.983 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 1 librados: starting msgr at 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 1 librados: starting objecter 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 1 Processor -- start 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 1 -- start start 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 1 librados: setting wanted keys 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 1 librados: calling monclient init 2024-03-25T14:29:16.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.977+0000 7f0f9c415640 10 monclient: init 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 10 monclient(hunting): picked mon.noname-c con 0x7f0f940a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9413f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 10 monclient(hunting): picked mon.noname-a con 0x7f0f941430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94137ab0 con 0x7f0f941430c0 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94138bd0 con 0x7f0f9413f710 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94138e40 con 0x7f0f940a98f0 2024-03-25T14:29:16.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:16.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.978+0000 7f0f9c415640 10 monclient(hunting): authenticate will time out at 1920.123385s 2024-03-25T14:29:16.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.994+0000 7f0f9a9b2640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f0f941430c0 0x7f0f941e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:60536/0 (socket says 172.21.15.63:60536) 2024-03-25T14:29:16.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.994+0000 7f0f9a9b2640 1 -- 172.21.15.63:0/2417148317 learned_addr learned my addr 172.21.15.63:0/2417148317 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:17.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.996+0000 7f0f8affd640 10 client.?.objecter ms_handle_connect 0x7f0f9413f710 2024-03-25T14:29:17.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.997+0000 7f0f8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:17.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.999+0000 7f0f8affd640 10 client.?.objecter ms_handle_connect 0x7f0f941430c0 2024-03-25T14:29:17.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:16.999+0000 7f0f8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:17.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.003+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750560544 0 0) 0x7f0f94138bd0 con 0x7f0f9413f710 2024-03-25T14:29:17.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.003+0000 7f0f8affd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:17.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.003+0000 7f0f8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:17.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.003+0000 7f0f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:17.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.003+0000 7f0f8affd640 10 monclient(hunting): my global_id is 4654 2024-03-25T14:29:17.008 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.003+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f78004eb0 con 0x7f0f9413f710 2024-03-25T14:29:17.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.008+0000 7f0f8affd640 10 client.?.objecter ms_handle_connect 0x7f0f940a98f0 2024-03-25T14:29:17.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.008+0000 7f0f8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:17.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.018+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762717527 0 0) 0x7f0f94137ab0 con 0x7f0f941430c0 2024-03-25T14:29:17.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.018+0000 7f0f8affd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:17.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.018+0000 7f0f8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:17.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.018+0000 7f0f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:17.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.018+0000 7f0f8affd640 10 monclient(hunting): my global_id is 4650 2024-03-25T14:29:17.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.018+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f94138bd0 con 0x7f0f941430c0 2024-03-25T14:29:17.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.021+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3164308342 0 0) 0x7f0f78004eb0 con 0x7f0f9413f710 2024-03-25T14:29:17.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.021+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f94137ab0 con 0x7f0f9413f710 2024-03-25T14:29:17.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.021+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f840018f0 con 0x7f0f9413f710 2024-03-25T14:29:17.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.021+0000 7f0f8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.021+0000 7f0f8affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.021+0000 7f0f8affd640 10 monclient(hunting): dump: 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:17.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.021+0000 7f0f8affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:17.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.022+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736665733 0 0) 0x7f0f94138e40 con 0x7f0f940a98f0 2024-03-25T14:29:17.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.022+0000 7f0f8affd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:17.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.022+0000 7f0f8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:17.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.022+0000 7f0f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:17.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.022+0000 7f0f8affd640 10 monclient(hunting): my global_id is 4628 2024-03-25T14:29:17.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.022+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f78004eb0 con 0x7f0f940a98f0 2024-03-25T14:29:17.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 317952626 0 0) 0x7f0f94137ab0 con 0x7f0f9413f710 2024-03-25T14:29:17.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 >> v1:172.21.15.63:6790/0 conn(0x7f0f940a98f0 legacy=0x7f0f94137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:17.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 >> v1:172.21.15.63:6789/0 conn(0x7f0f941430c0 legacy=0x7f0f941e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:17.036 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 10 monclient: _finish_hunting 0 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 1 monclient: found mon.b 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94139120 con 0x7f0f9413f710 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 10 monclient: _finish_auth 0 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 10 monclient: _check_auth_tickets 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.031+0000 7f0f8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 5 monclient: authenticate success, global_id 4654 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 10 monclient: _renew_subs 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f94139400 con 0x7f0f9413f710 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 10 monclient: _renew_subs 2024-03-25T14:29:17.037 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:17.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f940b7880 con 0x7f0f9413f710 2024-03-25T14:29:17.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.032+0000 7f0f9c415640 1 librados: init done 2024-03-25T14:29:17.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.033+0000 7f0f9c415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:17.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.033+0000 7f0f9c415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:17.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.033+0000 7f0f9c415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:17.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.033+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f94137810 con 0x7f0f9413f710 2024-03-25T14:29:17.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.312+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f84003610 con 0x7f0f9413f710 2024-03-25T14:29:17.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.312+0000 7f0f8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:17.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.312+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0f840039a0 con 0x7f0f9413f710 2024-03-25T14:29:17.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.312+0000 7f0f8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:17.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.312+0000 7f0f8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:17.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.312+0000 7f0f8affd640 10 monclient: dump: 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:17.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.312+0000 7f0f989ae640 4 set_mon_vals no callback set 2024-03-25T14:29:17.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.313+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f0f84003bc0 con 0x7f0f9413f710 2024-03-25T14:29:17.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.313+0000 7f0f9a1b1640 10 monclient: get_auth_request con 0x7f0f7806de00 auth_method 0 2024-03-25T14:29:17.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.314+0000 7f0f8affd640 10 client.4654.objecter ms_handle_connect 0x7f0f7806de00 2024-03-25T14:29:17.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f0f84053600 con 0x7f0f9413f710 2024-03-25T14:29:17.322 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 10 client.4654.objecter ms_dispatch 0x7f0f94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:17.323 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 3 client.4654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:17.323 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 3 client.4654.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:17.323 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T14:29:17.323 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f8404f110 con 0x7f0f9413f710 2024-03-25T14:29:17.323 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:17.323 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.318+0000 7f0f8affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:17.543 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.539+0000 7f0f9c415640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:17.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.539+0000 7f0f9c415640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:17.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.539+0000 7f0f9c415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:17.544 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.539+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f94137c60 con 0x7f0f9413f710 2024-03-25T14:29:17.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.585+0000 7f0f8affd640 1 -- 172.21.15.63:0/2417148317 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f0f94137c60 con 0x7f0f9413f710 2024-03-25T14:29:17.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.585+0000 7f0f8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:17.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.585+0000 7f0f8affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:17.590 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:17.591 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:17.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.587+0000 7f0f9c415640 10 librados: watch_flush enter 2024-03-25T14:29:17.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 10 librados: watch_flush exit 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 10 client.4654.objecter successfully canceled tick 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 >> v1:172.21.15.153:6800/1688255552 conn(0x7f0f7806de00 legacy=0x7f0f780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 10 monclient: shutdown 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 >> v1:172.21.15.153:6789/0 conn(0x7f0f9413f710 legacy=0x7f0f941df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 shutdown_connections 2024-03-25T14:29:17.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 >> 172.21.15.63:0/2417148317 conn(0x7f0f940b7ea0 msgr2=0x7f0f94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:17.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 shutdown_connections 2024-03-25T14:29:17.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 1 -- 172.21.15.63:0/2417148317 wait complete. 2024-03-25T14:29:17.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:17.588+0000 7f0f9c415640 1 librados: shutdown 2024-03-25T14:29:17.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:24.412 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:24.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.522+0000 7f190a1a4640 10 monclient: get_monmap_and_config 2024-03-25T14:29:24.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.522+0000 7f190a1a4640 10 monclient: build_initial_monmap 2024-03-25T14:29:24.527 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.522+0000 7f190a1a4640 10 monclient: monmap: 2024-03-25T14:29:24.527 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:24.527 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:24.527 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:24.523755+0000 2024-03-25T14:29:24.528 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:24.523755+0000 2024-03-25T14:29:24.528 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:24.528 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:24.528 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:24.528 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:24.528 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:24.528 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.523+0000 7f190a1a4640 1 Processor -- start 2024-03-25T14:29:24.529 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.523+0000 7f190a1a4640 1 -- start start 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.523+0000 7f190a1a4640 10 monclient: init 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 10 monclient(hunting): picked mon.noname-a con 0x7f19040b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 10 monclient(hunting): picked mon.noname-c con 0x7f19040b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 10 monclient(hunting): picked mon.noname-b con 0x7f190414be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19040af940 con 0x7f19040b4d60 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19040b0a70 con 0x7f19040b1730 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19040b0d50 con 0x7f190414be90 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:24.530 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.524+0000 7f190a1a4640 10 monclient(hunting): authenticate will time out at 1927.669088s 2024-03-25T14:29:24.549 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.545+0000 7f1902ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f19040b1730 0x7f19040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:33784/0 (socket says 172.21.15.63:33784) 2024-03-25T14:29:24.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.545+0000 7f1902ffd640 1 -- 172.21.15.63:0/216038398 learned_addr learned my addr 172.21.15.63:0/216038398 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:24.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.556+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178939581 0 0) 0x7f19040b0a70 con 0x7f19040b1730 2024-03-25T14:29:24.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.556+0000 7f19027fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:24.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.556+0000 7f19027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:24.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.556+0000 7f19027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:24.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.556+0000 7f19027fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T14:29:24.561 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.556+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ec003520 con 0x7f19040b1730 2024-03-25T14:29:24.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.561+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791518306 0 0) 0x7f19040af940 con 0x7f19040b4d60 2024-03-25T14:29:24.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.561+0000 7f19027fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:24.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.561+0000 7f19027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:24.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.561+0000 7f19027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:24.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.561+0000 7f19027fc640 10 monclient(hunting): my global_id is 4653 2024-03-25T14:29:24.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.561+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19040b0a70 con 0x7f19040b4d60 2024-03-25T14:29:24.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.569+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522087057 0 0) 0x7f19040b0d50 con 0x7f190414be90 2024-03-25T14:29:24.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.569+0000 7f19027fc640 10 monclient(hunting): _negotiate 2024-03-25T14:29:24.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.569+0000 7f19027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:24.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.569+0000 7f19027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:24.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.569+0000 7f19027fc640 10 monclient(hunting): my global_id is 4631 2024-03-25T14:29:24.574 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.569+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19040af940 con 0x7f190414be90 2024-03-25T14:29:24.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3199670706 0 0) 0x7f19040b0a70 con 0x7f19040b4d60 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 >> v1:172.21.15.63:6790/0 conn(0x7f190414be90 legacy=0x7f190414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 >> v1:172.21.15.153:6789/0 conn(0x7f19040b1730 legacy=0x7f19040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 10 monclient: _finish_hunting 0 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 1 monclient: found mon.noname-a 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19040b1030 con 0x7f19040b4d60 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 10 monclient: _finish_auth 0 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 10 monclient: _check_auth_tickets 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:24.593 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f18f4001670 con 0x7f19040b4d60 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f19027fc640 10 monclient: dump: 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:24.594 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f190a1a4640 5 monclient: authenticate success, global_id 4653 2024-03-25T14:29:24.595 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.587+0000 7f190a1a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18f40017d0 con 0x7f19040b4d60 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f19027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f19027fc640 1 -- 172.21.15.63:0/216038398 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f18f4001be0 con 0x7f19040b4d60 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f19027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f19027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f19027fc640 10 monclient: dump: 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:24.599 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f190a1a4640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:24.600 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.594+0000 7f190a1a4640 4 set_mon_vals no callback set 2024-03-25T14:29:24.601 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.595+0000 7f190a1a4640 10 monclient: shutdown 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.595+0000 7f190a1a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.595+0000 7f190a1a4640 1 -- 172.21.15.63:0/216038398 >> v1:172.21.15.63:6789/0 conn(0x7f19040b4d60 legacy=0x7f190413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.595+0000 7f190a1a4640 1 -- 172.21.15.63:0/216038398 shutdown_connections 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.595+0000 7f190a1a4640 1 -- 172.21.15.63:0/216038398 >> 172.21.15.63:0/216038398 conn(0x7f19040617a0 msgr2=0x7f1904061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.595+0000 7f190a1a4640 1 -- 172.21.15.63:0/216038398 shutdown_connections 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.595+0000 7f190a1a4640 1 -- 172.21.15.63:0/216038398 wait complete. 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.596+0000 7f190a1a4640 10 monclient: build_initial_monmap 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.596+0000 7f190a1a4640 10 monclient: monmap: 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:24.602 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.596+0000 7f190a1a4640 1 librados: starting msgr at 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.596+0000 7f190a1a4640 1 librados: starting objecter 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.597+0000 7f190a1a4640 1 Processor -- start 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.597+0000 7f190a1a4640 1 -- start start 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.597+0000 7f190a1a4640 1 librados: setting wanted keys 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.597+0000 7f190a1a4640 1 librados: calling monclient init 2024-03-25T14:29:24.603 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.597+0000 7f190a1a4640 10 monclient: init 2024-03-25T14:29:24.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:24.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:29:24.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 10 monclient(hunting): picked mon.noname-c con 0x7f19040b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:24.604 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 10 monclient(hunting): picked mon.noname-a con 0x7f19040b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:24.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 10 monclient(hunting): picked mon.noname-b con 0x7f190414be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:24.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1904143d40 con 0x7f19040b4d60 2024-03-25T14:29:24.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1904140f00 con 0x7f190414be90 2024-03-25T14:29:24.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1904138460 con 0x7f19040b1730 2024-03-25T14:29:24.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:24.605 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.598+0000 7f190a1a4640 10 monclient(hunting): authenticate will time out at 1927.742960s 2024-03-25T14:29:24.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.615+0000 7f1903fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f190414be90 0x7f19041d7d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:33836/0 (socket says 172.21.15.63:33836) 2024-03-25T14:29:24.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.615+0000 7f1903fff640 1 -- 172.21.15.63:0/1887872199 learned_addr learned my addr 172.21.15.63:0/1887872199 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:24.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.622+0000 7f18ebfff640 10 client.?.objecter ms_handle_connect 0x7f19040b1730 2024-03-25T14:29:24.627 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.622+0000 7f18ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:24.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.627+0000 7f18ebfff640 10 client.?.objecter ms_handle_connect 0x7f19040b4d60 2024-03-25T14:29:24.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.627+0000 7f18ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:24.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.628+0000 7f18ebfff640 10 client.?.objecter ms_handle_connect 0x7f190414be90 2024-03-25T14:29:24.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.628+0000 7f18ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:24.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.637+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766277088 0 0) 0x7f1904138460 con 0x7f19040b1730 2024-03-25T14:29:24.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.637+0000 7f18ebfff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:24.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.637+0000 7f18ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:24.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.637+0000 7f18ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:24.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.637+0000 7f18ebfff640 10 monclient(hunting): my global_id is 4634 2024-03-25T14:29:24.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.637+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f0004eb0 con 0x7f19040b1730 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.640+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222537297 0 0) 0x7f1904140f00 con 0x7f190414be90 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.640+0000 7f18ebfff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.640+0000 7f18ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.640+0000 7f18ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.640+0000 7f18ebfff640 10 monclient(hunting): my global_id is 4660 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.640+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1904138460 con 0x7f190414be90 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.641+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255546169 0 0) 0x7f1904143d40 con 0x7f19040b4d60 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.641+0000 7f18ebfff640 10 monclient(hunting): _negotiate 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.641+0000 7f18ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.641+0000 7f18ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.641+0000 7f18ebfff640 10 monclient(hunting): my global_id is 4656 2024-03-25T14:29:24.646 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.641+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1904140f00 con 0x7f19040b4d60 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.648+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3297908701 0 0) 0x7f18f0004eb0 con 0x7f19040b1730 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.648+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1904143d40 con 0x7f19040b1730 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.648+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f18f4001ba0 con 0x7f19040b1730 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.648+0000 7f18ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.648+0000 7f18ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.648+0000 7f18ebfff640 10 monclient(hunting): dump: 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.653 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:24.654 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:24.654 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:24.654 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:24.654 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:24.654 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.648+0000 7f18ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.665+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 919477817 0 0) 0x7f1904138460 con 0x7f190414be90 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.665+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18f0004eb0 con 0x7f190414be90 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.665+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f18fc001c50 con 0x7f190414be90 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.665+0000 7f18ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.665+0000 7f18ebfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.665+0000 7f18ebfff640 10 monclient(hunting): dump: 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:24.670 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:24.671 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.667+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4166212730 0 0) 0x7f1904140f00 con 0x7f19040b4d60 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.667+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1904138460 con 0x7f19040b4d60 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.667+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f18f80017b0 con 0x7f19040b4d60 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.667+0000 7f18ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.667+0000 7f18ebfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.667+0000 7f18ebfff640 10 monclient(hunting): dump: 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:24.672 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.673 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.673 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:24.673 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:24.673 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:24.673 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:24.673 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:24.673 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2190474501 0 0) 0x7f1904143d40 con 0x7f19040b1730 2024-03-25T14:29:24.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 >> v1:172.21.15.153:6789/0 conn(0x7f190414be90 legacy=0x7f19041d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:24.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 >> v1:172.21.15.63:6789/0 conn(0x7f19040b4d60 legacy=0x7f1904142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:24.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 10 monclient: _finish_hunting 0 2024-03-25T14:29:24.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 1 monclient: found mon.c 2024-03-25T14:29:24.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1904141130 con 0x7f19040b1730 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 10 monclient: _finish_auth 0 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 10 monclient: _check_auth_tickets 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f18ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 5 monclient: authenticate success, global_id 4634 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 10 monclient: _renew_subs 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1904137660 con 0x7f19040b1730 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 10 monclient: _renew_subs 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1904143030 con 0x7f19040b1730 2024-03-25T14:29:24.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.670+0000 7f190a1a4640 1 librados: init done 2024-03-25T14:29:24.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.671+0000 7f190a1a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:24.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.671+0000 7f190a1a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:24.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.671+0000 7f190a1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:24.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.671+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19040627b0 con 0x7f19040b1730 2024-03-25T14:29:24.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.993+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18f40038a0 con 0x7f19040b1730 2024-03-25T14:29:24.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.993+0000 7f18ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:24.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.993+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f18f4003bb0 con 0x7f19040b1730 2024-03-25T14:29:24.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.993+0000 7f18ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:24.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.993+0000 7f18ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:24.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.993+0000 7f18ebfff640 10 monclient: dump: 2024-03-25T14:29:24.998 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:24.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.993+0000 7f1901ffb640 4 set_mon_vals no callback set 2024-03-25T14:29:25.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.994+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f18f4019b80 con 0x7f19040b1730 2024-03-25T14:29:25.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.994+0000 7f1902ffd640 10 monclient: get_auth_request con 0x7f18f006e0b0 auth_method 0 2024-03-25T14:29:25.000 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:24.994+0000 7f18ebfff640 10 client.4634.objecter ms_handle_connect 0x7f18f006e0b0 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f18f40536d0 con 0x7f19040b1730 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 10 client.4634.objecter ms_dispatch 0x7f1904000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18f404f1e0 con 0x7f19040b1730 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:25.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.000+0000 7f18ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:25.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.249+0000 7f190a1a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:25.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.249+0000 7f190a1a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:25.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.249+0000 7f190a1a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:25.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.249+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19041d7980 con 0x7f19040b1730 2024-03-25T14:29:25.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.420+0000 7f18ebfff640 1 -- 172.21.15.63:0/1887872199 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f18f4051110 con 0x7f19040b1730 2024-03-25T14:29:25.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.420+0000 7f18ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:25.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.420+0000 7f18ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:25.425 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:25.425 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:25.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 10 librados: watch_flush enter 2024-03-25T14:29:25.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 10 librados: watch_flush exit 2024-03-25T14:29:25.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T14:29:25.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 10 client.4634.objecter successfully canceled tick 2024-03-25T14:29:25.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 >> v1:172.21.15.153:6800/1688255552 conn(0x7f18f006e0b0 legacy=0x7f18f0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.427 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 10 monclient: shutdown 2024-03-25T14:29:25.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:25.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 >> v1:172.21.15.63:6790/0 conn(0x7f19040b1730 legacy=0x7f1904137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:25.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 shutdown_connections 2024-03-25T14:29:25.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 >> 172.21.15.63:0/1887872199 conn(0x7f19040617a0 msgr2=0x7f19040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:25.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 shutdown_connections 2024-03-25T14:29:25.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 1 -- 172.21.15.63:0/1887872199 wait complete. 2024-03-25T14:29:25.428 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:25.422+0000 7f190a1a4640 1 librados: shutdown 2024-03-25T14:29:25.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:32.247 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:32.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.355+0000 7f44a2967640 10 monclient: get_monmap_and_config 2024-03-25T14:29:32.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.355+0000 7f44a2967640 10 monclient: build_initial_monmap 2024-03-25T14:29:32.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.355+0000 7f44a2967640 10 monclient: monmap: 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:32.356919+0000 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:32.356919+0000 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:32.361 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.356+0000 7f44a2967640 1 Processor -- start 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.356+0000 7f44a2967640 1 -- start start 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.356+0000 7f44a2967640 10 monclient: init 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 10 monclient(hunting): picked mon.noname-c con 0x7f449c0b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 10 monclient(hunting): picked mon.noname-b con 0x7f449c0b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 10 monclient(hunting): picked mon.noname-a con 0x7f449c14be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0af940 con 0x7f449c14be90 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0b0a70 con 0x7f449c0b4d60 2024-03-25T14:29:32.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0b0d50 con 0x7f449c0b1730 2024-03-25T14:29:32.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:32.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.357+0000 7f44a2967640 10 monclient(hunting): authenticate will time out at 1935.501927s 2024-03-25T14:29:32.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.366+0000 7f44a0f04640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f449c14be90 0x7f449c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51360/0 (socket says 172.21.15.63:51360) 2024-03-25T14:29:32.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.366+0000 7f44a0f04640 1 -- 172.21.15.63:0/3501079127 learned_addr learned my addr 172.21.15.63:0/3501079127 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:32.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.384+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996197760 0 0) 0x7f449c0af940 con 0x7f449c14be90 2024-03-25T14:29:32.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.384+0000 7f449affd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.384+0000 7f449affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.384+0000 7f449affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.384+0000 7f449affd640 10 monclient(hunting): my global_id is 4659 2024-03-25T14:29:32.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.384+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4484003520 con 0x7f449c14be90 2024-03-25T14:29:32.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.393+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966839358 0 0) 0x7f449c0b0a70 con 0x7f449c0b4d60 2024-03-25T14:29:32.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.393+0000 7f449affd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.393+0000 7f449affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.393+0000 7f449affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.393+0000 7f449affd640 10 monclient(hunting): my global_id is 4663 2024-03-25T14:29:32.399 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.393+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f449c0af940 con 0x7f449c0b4d60 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244692667 0 0) 0x7f4484003520 con 0x7f449c14be90 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 >> v1:172.21.15.63:6790/0 conn(0x7f449c0b1730 legacy=0x7f449c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 >> v1:172.21.15.153:6789/0 conn(0x7f449c0b4d60 legacy=0x7f449c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 10 monclient: _finish_hunting 0 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 1 monclient: found mon.noname-a 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:29:32.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c0b1030 con 0x7f449c14be90 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 10 monclient: _finish_auth 0 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 10 monclient: _check_auth_tickets 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4494001a10 con 0x7f449c14be90 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f44a2967640 5 monclient: authenticate success, global_id 4659 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f44a2967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f449affd640 10 monclient: dump: 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.401 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:32.402 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:32.402 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:32.402 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:32.402 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:32.402 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:32.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.395+0000 7f44a2967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4494001b70 con 0x7f449c14be90 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f449affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f449affd640 1 -- 172.21.15.63:0/3501079127 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4494001f80 con 0x7f449c14be90 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f44a2967640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f44a2967640 4 set_mon_vals no callback set 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f449affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f449affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.407+0000 7f449affd640 10 monclient: dump: 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:32.412 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:32.413 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:32.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.408+0000 7f44a2967640 10 monclient: shutdown 2024-03-25T14:29:32.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.408+0000 7f44a2967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:32.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.408+0000 7f44a2967640 1 -- 172.21.15.63:0/3501079127 >> v1:172.21.15.63:6789/0 conn(0x7f449c14be90 legacy=0x7f449c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.408+0000 7f44a2967640 1 -- 172.21.15.63:0/3501079127 shutdown_connections 2024-03-25T14:29:32.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.408+0000 7f44a2967640 1 -- 172.21.15.63:0/3501079127 >> 172.21.15.63:0/3501079127 conn(0x7f449c0617a0 msgr2=0x7f449c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:32.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.408+0000 7f44a2967640 1 -- 172.21.15.63:0/3501079127 shutdown_connections 2024-03-25T14:29:32.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.408+0000 7f44a2967640 1 -- 172.21.15.63:0/3501079127 wait complete. 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.409+0000 7f44a2967640 10 monclient: build_initial_monmap 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.409+0000 7f44a2967640 10 monclient: monmap: 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:32.415 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:32.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.410+0000 7f44a2967640 1 librados: starting msgr at 2024-03-25T14:29:32.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.410+0000 7f44a2967640 1 librados: starting objecter 2024-03-25T14:29:32.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.410+0000 7f44a2967640 1 Processor -- start 2024-03-25T14:29:32.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.410+0000 7f44a2967640 1 -- start start 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.410+0000 7f44a2967640 1 librados: setting wanted keys 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.410+0000 7f44a2967640 1 librados: calling monclient init 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.410+0000 7f44a2967640 10 monclient: init 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 10 monclient(hunting): picked mon.noname-a con 0x7f449c0b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 10 monclient(hunting): picked mon.noname-c con 0x7f449c0b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 10 monclient(hunting): picked mon.noname-b con 0x7f449c14be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c1d87c0 con 0x7f449c0b1730 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c138820 con 0x7f449c14be90 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c1389b0 con 0x7f449c0b4d60 2024-03-25T14:29:32.418 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:32.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.411+0000 7f44a2967640 10 monclient(hunting): authenticate will time out at 1935.556099s 2024-03-25T14:29:32.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.418+0000 7f44a0f04640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f449c14be90 0x7f449c1e22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:46902/0 (socket says 172.21.15.63:46902) 2024-03-25T14:29:32.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.418+0000 7f44a0f04640 1 -- 172.21.15.63:0/691769356 learned_addr learned my addr 172.21.15.63:0/691769356 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:32.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.419+0000 7f4498ff9640 10 client.?.objecter ms_handle_connect 0x7f449c14be90 2024-03-25T14:29:32.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.419+0000 7f4498ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:32.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.421+0000 7f4498ff9640 10 client.?.objecter ms_handle_connect 0x7f449c0b1730 2024-03-25T14:29:32.426 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.421+0000 7f4498ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:32.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.426+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854926019 0 0) 0x7f449c138820 con 0x7f449c14be90 2024-03-25T14:29:32.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.426+0000 7f4498ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.430 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.426+0000 7f4498ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.426+0000 7f4498ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.426+0000 7f4498ff9640 10 monclient(hunting): my global_id is 4666 2024-03-25T14:29:32.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.426+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4488004eb0 con 0x7f449c14be90 2024-03-25T14:29:32.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.441+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564460480 0 0) 0x7f449c1d87c0 con 0x7f449c0b1730 2024-03-25T14:29:32.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.441+0000 7f4498ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:29:32.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.441+0000 7f4498ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:32.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.441+0000 7f4498ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:32.446 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.441+0000 7f4498ff9640 10 monclient(hunting): my global_id is 4662 2024-03-25T14:29:32.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.441+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f449c138820 con 0x7f449c0b1730 2024-03-25T14:29:32.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.442+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3061276278 0 0) 0x7f4488004eb0 con 0x7f449c14be90 2024-03-25T14:29:32.447 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.443+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f449c1d87c0 con 0x7f449c14be90 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.443+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4494001ec0 con 0x7f449c14be90 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.443+0000 7f4498ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.443+0000 7f4498ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.443+0000 7f4498ff9640 10 monclient(hunting): dump: 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:32.448 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:32.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.443+0000 7f4498ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:32.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.444+0000 7f4498ff9640 10 client.?.objecter ms_handle_connect 0x7f449c0b4d60 2024-03-25T14:29:32.449 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.444+0000 7f4498ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:32.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1474147369 0 0) 0x7f449c1d87c0 con 0x7f449c14be90 2024-03-25T14:29:32.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 >> v1:172.21.15.63:6790/0 conn(0x7f449c0b4d60 legacy=0x7f449c1d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.464 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 >> v1:172.21.15.63:6789/0 conn(0x7f449c0b1730 legacy=0x7f449c138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 1 monclient: found mon.b 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c1405f0 con 0x7f449c14be90 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 10 monclient: _finish_auth 0 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f4498ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 5 monclient: authenticate success, global_id 4666 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 10 monclient: _renew_subs 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c1407e0 con 0x7f449c14be90 2024-03-25T14:29:32.465 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:32.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 10 monclient: _renew_subs 2024-03-25T14:29:32.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:32.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c1e3ba0 con 0x7f449c14be90 2024-03-25T14:29:32.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.459+0000 7f44a2967640 1 librados: init done 2024-03-25T14:29:32.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.460+0000 7f44a2967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:32.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.460+0000 7f44a2967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:32.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.460+0000 7f44a2967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:32.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.460+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c1d7f00 con 0x7f449c14be90 2024-03-25T14:29:32.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.758+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4494004e10 con 0x7f449c14be90 2024-03-25T14:29:32.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.758+0000 7f4498ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:32.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.758+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f4494002100 con 0x7f449c14be90 2024-03-25T14:29:32.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.758+0000 7f4498ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:32.763 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.758+0000 7f4498ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.758+0000 7f4498ff9640 10 monclient: dump: 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:32.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.758+0000 7f449a7fc640 4 set_mon_vals no callback set 2024-03-25T14:29:32.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.759+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f449401af90 con 0x7f449c14be90 2024-03-25T14:29:32.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.760+0000 7f449bfff640 10 monclient: get_auth_request con 0x7f448806dcc0 auth_method 0 2024-03-25T14:29:32.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.760+0000 7f4498ff9640 10 client.4666.objecter ms_handle_connect 0x7f448806dcc0 2024-03-25T14:29:32.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7f4494054b00 con 0x7f449c14be90 2024-03-25T14:29:32.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 10 client.4666.objecter ms_dispatch 0x7f449c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:32.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:32.768 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:32.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T14:29:32.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4494050440 con 0x7f449c14be90 2024-03-25T14:29:32.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:32.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.763+0000 7f4498ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:32.821 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:11.333 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.872 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:11.384 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.909 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:11.420 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.944 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:11.455 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.952 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:11.411 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.955 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:11.467 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.959 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:11.418 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:32.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.968+0000 7f44a2967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:32.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.968+0000 7f44a2967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:32.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.968+0000 7f44a2967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:32.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:32.968+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f449c0b5ac0 con 0x7f449c14be90 2024-03-25T14:29:32.988 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:11.499 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:33.004 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:11.464 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:33.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.021+0000 7f4498ff9640 1 -- 172.21.15.63:0/691769356 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7f4494052540 con 0x7f449c14be90 2024-03-25T14:29:33.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.021+0000 7f4498ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:33.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.021+0000 7f4498ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:33.026 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:33.026 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:33.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 10 librados: watch_flush enter 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 10 librados: watch_flush exit 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 10 client.4666.objecter successfully canceled tick 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 >> v1:172.21.15.153:6800/1688255552 conn(0x7f448806dcc0 legacy=0x7f4488070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 10 monclient: shutdown 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 >> v1:172.21.15.153:6789/0 conn(0x7f449c14be90 legacy=0x7f449c1e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 shutdown_connections 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 >> 172.21.15.63:0/691769356 conn(0x7f449c0617a0 msgr2=0x7f449c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 shutdown_connections 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 1 -- 172.21.15.63:0/691769356 wait complete. 2024-03-25T14:29:33.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:33.023+0000 7f44a2967640 1 librados: shutdown 2024-03-25T14:29:33.039 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:39.847 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:39.965 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.961+0000 7fdd23196640 10 monclient: get_monmap_and_config 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.961+0000 7fdd23196640 10 monclient: build_initial_monmap 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.961+0000 7fdd23196640 10 monclient: monmap: 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:39.962180+0000 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:39.962180+0000 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:39.966 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:39.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.962+0000 7fdd23196640 1 Processor -- start 2024-03-25T14:29:39.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.962+0000 7fdd23196640 1 -- start start 2024-03-25T14:29:39.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.962+0000 7fdd23196640 10 monclient: init 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c142520 con 0x7fdd1c1430c0 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c0b28a0 con 0x7fdd1c0a98f0 2024-03-25T14:29:39.968 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c0b2a30 con 0x7fdd1c13f710 2024-03-25T14:29:39.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:39.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.963+0000 7fdd23196640 10 monclient(hunting): authenticate will time out at 1943.107864s 2024-03-25T14:29:39.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.971+0000 7fdd20f32640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdd1c0a98f0 0x7fdd1c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:50574/0 (socket says 172.21.15.63:50574) 2024-03-25T14:29:39.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.971+0000 7fdd20f32640 1 -- 172.21.15.63:0/3843610029 learned_addr learned my addr 172.21.15.63:0/3843610029 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:39.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.975+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409922677 0 0) 0x7fdd1c0b28a0 con 0x7fdd1c0a98f0 2024-03-25T14:29:39.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.975+0000 7fdd1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:29:39.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.975+0000 7fdd1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:39.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.975+0000 7fdd1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:39.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.975+0000 7fdd1b7fe640 10 monclient(hunting): my global_id is 4669 2024-03-25T14:29:39.980 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.975+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd04003520 con 0x7fdd1c0a98f0 2024-03-25T14:29:39.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.980+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390203928 0 0) 0x7fdd1c142520 con 0x7fdd1c1430c0 2024-03-25T14:29:39.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.980+0000 7fdd1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:29:39.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.980+0000 7fdd1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:39.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.980+0000 7fdd1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:39.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.980+0000 7fdd1b7fe640 10 monclient(hunting): my global_id is 4665 2024-03-25T14:29:39.985 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.980+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd1c0b28a0 con 0x7fdd1c1430c0 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1370821057 0 0) 0x7fdd04003520 con 0x7fdd1c0a98f0 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 >> v1:172.21.15.63:6790/0 conn(0x7fdd1c13f710 legacy=0x7fdd1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 >> v1:172.21.15.63:6789/0 conn(0x7fdd1c1430c0 legacy=0x7fdd1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 1 monclient: found mon.noname-c 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c0b44c0 con 0x7fdd1c0a98f0 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 10 monclient: _finish_auth 0 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:39.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdd10001690 con 0x7fdd1c0a98f0 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd1b7fe640 10 monclient: dump: 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:39.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd23196640 5 monclient: authenticate success, global_id 4669 2024-03-25T14:29:39.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.988+0000 7fdd23196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd100017f0 con 0x7fdd1c0a98f0 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd1b7fe640 1 -- 172.21.15.63:0/3843610029 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdd10001c00 con 0x7fdd1c0a98f0 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd23196640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd23196640 4 set_mon_vals no callback set 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:40.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd1b7fe640 10 monclient: dump: 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:40.002 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:40.003 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:40.003 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:40.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd23196640 10 monclient: shutdown 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd23196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.996+0000 7fdd23196640 1 -- 172.21.15.63:0/3843610029 >> v1:172.21.15.153:6789/0 conn(0x7fdd1c0a98f0 legacy=0x7fdd1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.997+0000 7fdd23196640 1 -- 172.21.15.63:0/3843610029 shutdown_connections 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.997+0000 7fdd23196640 1 -- 172.21.15.63:0/3843610029 >> 172.21.15.63:0/3843610029 conn(0x7fdd1c0b7ea0 msgr2=0x7fdd1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.997+0000 7fdd23196640 1 -- 172.21.15.63:0/3843610029 shutdown_connections 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.997+0000 7fdd23196640 1 -- 172.21.15.63:0/3843610029 wait complete. 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.997+0000 7fdd23196640 10 monclient: build_initial_monmap 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.997+0000 7fdd23196640 10 monclient: monmap: 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:40.005 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.998+0000 7fdd23196640 1 librados: starting msgr at 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.998+0000 7fdd23196640 1 librados: starting objecter 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.998+0000 7fdd23196640 1 Processor -- start 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.998+0000 7fdd23196640 1 -- start start 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.998+0000 7fdd23196640 1 librados: setting wanted keys 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.998+0000 7fdd23196640 1 librados: calling monclient init 2024-03-25T14:29:40.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.998+0000 7fdd23196640 10 monclient: init 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c0a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1c13f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c1430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c139510 con 0x7fdd1c0a98f0 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c1366a0 con 0x7fdd1c13f710 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c1d7e80 con 0x7fdd1c1430c0 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:40.007 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:39.999+0000 7fdd23196640 10 monclient(hunting): authenticate will time out at 1943.143982s 2024-03-25T14:29:40.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.007+0000 7fdd1bfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdd1c13f710 0x7fdd1c0b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:50598/0 (socket says 172.21.15.63:50598) 2024-03-25T14:29:40.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.007+0000 7fdd1bfff640 1 -- 172.21.15.63:0/568913197 learned_addr learned my addr 172.21.15.63:0/568913197 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:40.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.011+0000 7fdd197fa640 10 client.?.objecter ms_handle_connect 0x7fdd1c13f710 2024-03-25T14:29:40.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.011+0000 7fdd197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:40.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163094945 0 0) 0x7fdd1c1366a0 con 0x7fdd1c13f710 2024-03-25T14:29:40.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 10 monclient(hunting): _negotiate 2024-03-25T14:29:40.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:40.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:40.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 10 monclient(hunting): my global_id is 4672 2024-03-25T14:29:40.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd08004eb0 con 0x7fdd1c13f710 2024-03-25T14:29:40.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 10 client.?.objecter ms_handle_connect 0x7fdd1c0a98f0 2024-03-25T14:29:40.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.016+0000 7fdd197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:40.023 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.019+0000 7fdd197fa640 10 client.?.objecter ms_handle_connect 0x7fdd1c1430c0 2024-03-25T14:29:40.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.019+0000 7fdd197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:40.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.024+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523251903 0 0) 0x7fdd08004eb0 con 0x7fdd1c13f710 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.024+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd1c1366a0 con 0x7fdd1c13f710 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.024+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdd14001ce0 con 0x7fdd1c13f710 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.024+0000 7fdd197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.024+0000 7fdd197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.024+0000 7fdd197fa640 10 monclient(hunting): dump: 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:40.029 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:40.030 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:40.030 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:40.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.024+0000 7fdd197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2388017986 0 0) 0x7fdd1c1366a0 con 0x7fdd1c13f710 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 >> v1:172.21.15.63:6790/0 conn(0x7fdd1c1430c0 legacy=0x7fdd1c1372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 >> v1:172.21.15.63:6789/0 conn(0x7fdd1c0a98f0 legacy=0x7fdd1c1d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 10 monclient: _finish_hunting 0 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 1 monclient: found mon.b 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c1d8010 con 0x7fdd1c13f710 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 10 monclient: _finish_auth 0 2024-03-25T14:29:40.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 10 monclient: _check_auth_tickets 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd23196640 5 monclient: authenticate success, global_id 4672 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd23196640 10 monclient: _renew_subs 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd23196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.033+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c14e5e0 con 0x7fdd1c13f710 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.034+0000 7fdd23196640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.034+0000 7fdd23196640 10 monclient: _renew_subs 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.034+0000 7fdd23196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.034+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c0b14b0 con 0x7fdd1c13f710 2024-03-25T14:29:40.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.034+0000 7fdd23196640 1 librados: init done 2024-03-25T14:29:40.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.035+0000 7fdd23196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:40.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.035+0000 7fdd23196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:40.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.035+0000 7fdd23196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:40.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.035+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c1d8440 con 0x7fdd1c13f710 2024-03-25T14:29:40.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.305+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd14003970 con 0x7fdd1c13f710 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.305+0000 7fdd197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.305+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fdd14003d80 con 0x7fdd1c13f710 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.305+0000 7fdd197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.305+0000 7fdd197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.305+0000 7fdd197fa640 10 monclient: dump: 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:40.310 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:40.311 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:40.311 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:40.311 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:40.311 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:40.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.305+0000 7fdd1affd640 4 set_mon_vals no callback set 2024-03-25T14:29:40.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.306+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fdd14003fa0 con 0x7fdd1c13f710 2024-03-25T14:29:40.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.306+0000 7fdd20f32640 10 monclient: get_auth_request con 0x7fdd0806dd00 auth_method 0 2024-03-25T14:29:40.312 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.307+0000 7fdd197fa640 10 client.4672.objecter ms_handle_connect 0x7fdd0806dd00 2024-03-25T14:29:40.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.313+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7fdd14054b70 con 0x7fdd1c13f710 2024-03-25T14:29:40.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.313+0000 7fdd197fa640 10 client.4672.objecter ms_dispatch 0x7fdd1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:40.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.313+0000 7fdd197fa640 3 client.4672.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:40.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.313+0000 7fdd197fa640 3 client.4672.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:40.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.313+0000 7fdd197fa640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T14:29:40.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.314+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd14050680 con 0x7fdd1c13f710 2024-03-25T14:29:40.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.314+0000 7fdd197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:40.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.314+0000 7fdd197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:40.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.587+0000 7fdd23196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:40.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.587+0000 7fdd23196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:40.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.587+0000 7fdd23196640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:40.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.587+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd1c0627b0 con 0x7fdd1c13f710 2024-03-25T14:29:40.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.606+0000 7fdd197fa640 1 -- 172.21.15.63:0/568913197 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7fdd140525b0 con 0x7fdd1c13f710 2024-03-25T14:29:40.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.606+0000 7fdd197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:40.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.606+0000 7fdd197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:40.612 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:40.612 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:40.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 10 librados: watch_flush enter 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 10 librados: watch_flush exit 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 10 client.4672.objecter successfully canceled tick 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 >> v1:172.21.15.153:6800/1688255552 conn(0x7fdd0806dd00 legacy=0x7fdd080701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 10 monclient: shutdown 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 >> v1:172.21.15.153:6789/0 conn(0x7fdd1c13f710 legacy=0x7fdd1c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 shutdown_connections 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 >> 172.21.15.63:0/568913197 conn(0x7fdd1c0b7ea0 msgr2=0x7fdd1c13d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 shutdown_connections 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 1 -- 172.21.15.63:0/568913197 wait complete. 2024-03-25T14:29:40.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:40.609+0000 7fdd23196640 1 librados: shutdown 2024-03-25T14:29:40.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:47.434 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:47.547 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.542+0000 7facd0967640 10 monclient: get_monmap_and_config 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.542+0000 7facd0967640 10 monclient: build_initial_monmap 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.542+0000 7facd0967640 10 monclient: monmap: 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:47.543915+0000 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:47.543915+0000 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:47.548 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:47.549 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.543+0000 7facd0967640 1 Processor -- start 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.543+0000 7facd0967640 1 -- start start 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.543+0000 7facd0967640 10 monclient: init 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 10 monclient(hunting): picked mon.noname-b con 0x7faccc0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 10 monclient(hunting): picked mon.noname-a con 0x7faccc13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 10 monclient(hunting): picked mon.noname-c con 0x7faccc1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:47.550 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc142520 con 0x7faccc13f710 2024-03-25T14:29:47.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0b28a0 con 0x7faccc1430c0 2024-03-25T14:29:47.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0b2a30 con 0x7faccc0a98f0 2024-03-25T14:29:47.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:47.551 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.544+0000 7facd0967640 10 monclient(hunting): authenticate will time out at 1950.689304s 2024-03-25T14:29:47.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.552+0000 7facc9d9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7faccc13f710 0x7faccc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:34066/0 (socket says 172.21.15.63:34066) 2024-03-25T14:29:47.557 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.552+0000 7facc9d9b640 1 -- 172.21.15.63:0/3076389581 learned_addr learned my addr 172.21.15.63:0/3076389581 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:47.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.561+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068084989 0 0) 0x7faccc142520 con 0x7faccc13f710 2024-03-25T14:29:47.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.562+0000 7facc959a640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.562+0000 7facc959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.562+0000 7facc959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.562+0000 7facc959a640 10 monclient(hunting): my global_id is 4671 2024-03-25T14:29:47.567 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.562+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb4003520 con 0x7faccc13f710 2024-03-25T14:29:47.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3236890600 0 0) 0x7facb4003520 con 0x7faccc13f710 2024-03-25T14:29:47.576 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 >> v1:172.21.15.63:6790/0 conn(0x7faccc0a98f0 legacy=0x7faccc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 >> v1:172.21.15.153:6789/0 conn(0x7faccc1430c0 legacy=0x7faccc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 10 monclient: _finish_hunting 0 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 1 monclient: found mon.noname-a 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0b44c0 con 0x7faccc13f710 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 10 monclient: _finish_auth 0 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 10 monclient: _check_auth_tickets 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7facc0001690 con 0x7faccc13f710 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.577 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facc959a640 10 monclient: dump: 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:47.578 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facd0967640 5 monclient: authenticate success, global_id 4671 2024-03-25T14:29:47.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.571+0000 7facd0967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:47.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc00017f0 con 0x7faccc13f710 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facc959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facc959a640 1 -- 172.21.15.63:0/3076389581 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7facc0001c00 con 0x7faccc13f710 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facc959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facc959a640 10 monclient: dump: 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.586 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facd0967640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:47.587 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.581+0000 7facd0967640 4 set_mon_vals no callback set 2024-03-25T14:29:47.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.582+0000 7facd0967640 10 monclient: shutdown 2024-03-25T14:29:47.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.582+0000 7facd0967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:47.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.582+0000 7facd0967640 1 -- 172.21.15.63:0/3076389581 >> v1:172.21.15.63:6789/0 conn(0x7faccc13f710 legacy=0x7faccc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.582+0000 7facd0967640 1 -- 172.21.15.63:0/3076389581 shutdown_connections 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.582+0000 7facd0967640 1 -- 172.21.15.63:0/3076389581 >> 172.21.15.63:0/3076389581 conn(0x7faccc0b7ea0 msgr2=0x7faccc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.582+0000 7facd0967640 1 -- 172.21.15.63:0/3076389581 shutdown_connections 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.582+0000 7facd0967640 1 -- 172.21.15.63:0/3076389581 wait complete. 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 10 monclient: build_initial_monmap 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 10 monclient: monmap: 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:47.590 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 1 librados: starting msgr at 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 1 librados: starting objecter 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 1 Processor -- start 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 1 -- start start 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 1 librados: setting wanted keys 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 1 librados: calling monclient init 2024-03-25T14:29:47.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.583+0000 7facd0967640 10 monclient: init 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 10 monclient(hunting): picked mon.noname-b con 0x7faccc0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 10 monclient(hunting): picked mon.noname-c con 0x7faccc13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 10 monclient(hunting): picked mon.noname-a con 0x7faccc1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0abd70 con 0x7faccc1430c0 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc1d7710 con 0x7faccc0a98f0 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc1d8340 con 0x7faccc13f710 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:47.592 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.584+0000 7facd0967640 10 monclient(hunting): authenticate will time out at 1950.729133s 2024-03-25T14:29:47.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.591+0000 7facca59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7faccc0a98f0 0x7faccc137be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:55304/0 (socket says 172.21.15.63:55304) 2024-03-25T14:29:47.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.591+0000 7facca59c640 1 -- 172.21.15.63:0/605874510 learned_addr learned my addr 172.21.15.63:0/605874510 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:47.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.593+0000 7facbaffd640 10 client.?.objecter ms_handle_connect 0x7faccc0a98f0 2024-03-25T14:29:47.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.593+0000 7facbaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:47.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.597+0000 7facbaffd640 10 client.?.objecter ms_handle_connect 0x7faccc1430c0 2024-03-25T14:29:47.602 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.597+0000 7facbaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:47.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.606+0000 7facbaffd640 10 client.?.objecter ms_handle_connect 0x7faccc13f710 2024-03-25T14:29:47.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.606+0000 7facbaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:47.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.610+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756996528 0 0) 0x7faccc1d7710 con 0x7faccc0a98f0 2024-03-25T14:29:47.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.610+0000 7facbaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.610+0000 7facbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.610+0000 7facbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.610+0000 7facbaffd640 10 monclient(hunting): my global_id is 4678 2024-03-25T14:29:47.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.610+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac004eb0 con 0x7faccc0a98f0 2024-03-25T14:29:47.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.615+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205513495 0 0) 0x7faccc0abd70 con 0x7faccc1430c0 2024-03-25T14:29:47.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.615+0000 7facbaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.615+0000 7facbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.615+0000 7facbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.615+0000 7facbaffd640 10 monclient(hunting): my global_id is 4674 2024-03-25T14:29:47.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.615+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faccc1d7710 con 0x7faccc1430c0 2024-03-25T14:29:47.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.620+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487898792 0 0) 0x7faccc1d8340 con 0x7faccc13f710 2024-03-25T14:29:47.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.620+0000 7facbaffd640 10 monclient(hunting): _negotiate 2024-03-25T14:29:47.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.620+0000 7facbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:47.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.620+0000 7facbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:47.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.620+0000 7facbaffd640 10 monclient(hunting): my global_id is 4649 2024-03-25T14:29:47.626 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.620+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faccc0abd70 con 0x7faccc13f710 2024-03-25T14:29:47.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.632+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 756644037 0 0) 0x7facac004eb0 con 0x7faccc0a98f0 2024-03-25T14:29:47.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.632+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faccc1d8340 con 0x7faccc0a98f0 2024-03-25T14:29:47.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.632+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7facbc001a30 con 0x7faccc0a98f0 2024-03-25T14:29:47.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.632+0000 7facbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.632+0000 7facbaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.632+0000 7facbaffd640 10 monclient(hunting): dump: 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.632+0000 7facbaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:47.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.639+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626633694 0 0) 0x7faccc0abd70 con 0x7faccc13f710 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.639+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facac004eb0 con 0x7faccc13f710 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.639+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7facc0001b00 con 0x7faccc13f710 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.639+0000 7facbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.639+0000 7facbaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.639+0000 7facbaffd640 10 monclient(hunting): dump: 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:47.644 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:47.645 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:47.645 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.651+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077086008 0 0) 0x7faccc1d7710 con 0x7faccc1430c0 2024-03-25T14:29:47.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.651+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faccc0abd70 con 0x7faccc1430c0 2024-03-25T14:29:47.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.651+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7facc4001b00 con 0x7faccc1430c0 2024-03-25T14:29:47.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.651+0000 7facbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.651+0000 7facbaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:47.656 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.651+0000 7facbaffd640 10 monclient(hunting): dump: 2024-03-25T14:29:47.656 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:47.657 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2202590008 0 0) 0x7facac004eb0 con 0x7faccc13f710 2024-03-25T14:29:47.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 >> v1:172.21.15.153:6789/0 conn(0x7faccc0a98f0 legacy=0x7faccc137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 >> v1:172.21.15.63:6789/0 conn(0x7faccc1430c0 legacy=0x7faccc1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:47.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 10 monclient: _finish_hunting 0 2024-03-25T14:29:47.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 1 monclient: found mon.c 2024-03-25T14:29:47.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc1d84d0 con 0x7faccc13f710 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 10 monclient: _finish_auth 0 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 10 monclient: _check_auth_tickets 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 5 monclient: authenticate success, global_id 4649 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 10 monclient: _renew_subs 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc1d41e0 con 0x7faccc13f710 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 10 monclient: _renew_subs 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:47.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc1d5150 con 0x7faccc13f710 2024-03-25T14:29:47.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.655+0000 7facd0967640 1 librados: init done 2024-03-25T14:29:47.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.656+0000 7facd0967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:47.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.656+0000 7facd0967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:47.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.656+0000 7facd0967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:47.663 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.656+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc1d7950 con 0x7faccc13f710 2024-03-25T14:29:47.982 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.977+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc0004a90 con 0x7faccc13f710 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.977+0000 7facbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.977+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7facc0004c00 con 0x7faccc13f710 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.977+0000 7facbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.977+0000 7facbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.977+0000 7facbaffd640 10 monclient: dump: 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:47.983 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:47.984 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:47.984 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:47.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.977+0000 7facc8d99640 4 set_mon_vals no callback set 2024-03-25T14:29:47.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.978+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7facc001abd0 con 0x7faccc13f710 2024-03-25T14:29:47.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.979+0000 7facca59c640 10 monclient: get_auth_request con 0x7facac06e110 auth_method 0 2024-03-25T14:29:47.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.979+0000 7facbaffd640 10 client.4649.objecter ms_handle_connect 0x7facac06e110 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.985+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 830821827 0 0) 0x7facc00546c0 con 0x7faccc13f710 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.985+0000 7facbaffd640 10 client.4649.objecter ms_dispatch 0x7faccc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.985+0000 7facbaffd640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.985+0000 7facbaffd640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.986+0000 7facbaffd640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.986+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facc00501d0 con 0x7faccc13f710 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.986+0000 7facbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:47.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:47.986+0000 7facbaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:48.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.249+0000 7facd0967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:48.254 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.249+0000 7facd0967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.249+0000 7facd0967640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:29:48.255 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.249+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faccc074cc0 con 0x7faccc13f710 2024-03-25T14:29:48.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.286+0000 7facbaffd640 1 -- 172.21.15.63:0/605874510 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4217382948) 0x7facc0052100 con 0x7faccc13f710 2024-03-25T14:29:48.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.286+0000 7facbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:48.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.286+0000 7facbaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:48.294 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:48.294 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:19:55.024565+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:19:55.024565+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:48.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.288+0000 7facd0967640 10 librados: watch_flush enter 2024-03-25T14:29:48.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 10 librados: watch_flush exit 2024-03-25T14:29:48.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T14:29:48.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 10 client.4649.objecter successfully canceled tick 2024-03-25T14:29:48.295 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 >> v1:172.21.15.153:6800/1688255552 conn(0x7facac06e110 legacy=0x7facac0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 10 monclient: shutdown 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 >> v1:172.21.15.63:6790/0 conn(0x7faccc13f710 legacy=0x7faccc1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 shutdown_connections 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 >> 172.21.15.63:0/605874510 conn(0x7faccc0b7ea0 msgr2=0x7faccc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 shutdown_connections 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 1 -- 172.21.15.63:0/605874510 wait complete. 2024-03-25T14:29:48.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:48.289+0000 7facd0967640 1 librados: shutdown 2024-03-25T14:29:48.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:29:53.048 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:31.560 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.100 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:31.612 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.131 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:31.642 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.148 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:31.607 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.154 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:31.613 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.177 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:31.636 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.179 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:31.691 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.186 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:31.698 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:53.204 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:31.716 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:29:55.114 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.228+0000 7fca7ce58640 10 monclient: get_monmap_and_config 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.228+0000 7fca7ce58640 10 monclient: build_initial_monmap 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.228+0000 7fca7ce58640 10 monclient: monmap: 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:29:55.228461+0000 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:29:55.228461+0000 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:55.232 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:55.233 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:55.233 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:29:55.233 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:29:55.233 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 1 Processor -- start 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 1 -- start start 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient: init 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient(hunting): picked mon.noname-a con 0x7fca780b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient(hunting): picked mon.noname-b con 0x7fca780b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient(hunting): picked mon.noname-c con 0x7fca7814be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:55.234 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780af940 con 0x7fca780b4d60 2024-03-25T14:29:55.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780b0a70 con 0x7fca7814be90 2024-03-25T14:29:55.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780b0d50 con 0x7fca780b1730 2024-03-25T14:29:55.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:55.235 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.229+0000 7fca7ce58640 10 monclient(hunting): authenticate will time out at 1958.373392s 2024-03-25T14:29:55.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.236+0000 7fca7659c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fca780b4d60 0x7fca7813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:40388/0 (socket says 172.21.15.63:40388) 2024-03-25T14:29:55.240 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.236+0000 7fca7659c640 1 -- 172.21.15.63:0/2555944890 learned_addr learned my addr 172.21.15.63:0/2555944890 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:55.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.253+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074194942 0 0) 0x7fca780af940 con 0x7fca780b4d60 2024-03-25T14:29:55.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.253+0000 7fca75d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.257 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.253+0000 7fca75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.253+0000 7fca75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.253+0000 7fca75d9b640 10 monclient(hunting): my global_id is 4677 2024-03-25T14:29:55.258 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.253+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64003520 con 0x7fca780b4d60 2024-03-25T14:29:55.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.262+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348793976 0 0) 0x7fca780b0a70 con 0x7fca7814be90 2024-03-25T14:29:55.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.262+0000 7fca75d9b640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.262+0000 7fca75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.262+0000 7fca75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.262+0000 7fca75d9b640 10 monclient(hunting): my global_id is 4681 2024-03-25T14:29:55.266 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.262+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca780af940 con 0x7fca7814be90 2024-03-25T14:29:55.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3760315557 0 0) 0x7fca64003520 con 0x7fca780b4d60 2024-03-25T14:29:55.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 >> v1:172.21.15.63:6790/0 conn(0x7fca780b1730 legacy=0x7fca780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 >> v1:172.21.15.153:6789/0 conn(0x7fca7814be90 legacy=0x7fca7814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.271 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 10 monclient: _finish_hunting 0 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 1 monclient: found mon.noname-a 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca780b1030 con 0x7fca780b4d60 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 10 monclient: _finish_auth 0 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 10 monclient: _check_auth_tickets 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca68001690 con 0x7fca780b4d60 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca75d9b640 10 monclient: dump: 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:55.272 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca7ce58640 5 monclient: authenticate success, global_id 4677 2024-03-25T14:29:55.273 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.267+0000 7fca7ce58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:29:55.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.274+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca680017f0 con 0x7fca780b4d60 2024-03-25T14:29:55.278 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca75d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca75d9b640 1 -- 172.21.15.63:0/2555944890 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca68001c00 con 0x7fca780b4d60 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca7ce58640 10 monclient: get_monmap_and_config success 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca7ce58640 4 set_mon_vals no callback set 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca75d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca75d9b640 10 monclient: dump: 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.279 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:55.280 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:55.280 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:55.280 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:55.280 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca7ce58640 10 monclient: shutdown 2024-03-25T14:29:55.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca7ce58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:55.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca7ce58640 1 -- 172.21.15.63:0/2555944890 >> v1:172.21.15.63:6789/0 conn(0x7fca780b4d60 legacy=0x7fca7813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca7ce58640 1 -- 172.21.15.63:0/2555944890 shutdown_connections 2024-03-25T14:29:55.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.275+0000 7fca7ce58640 1 -- 172.21.15.63:0/2555944890 >> 172.21.15.63:0/2555944890 conn(0x7fca780617a0 msgr2=0x7fca78061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:55.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.276+0000 7fca7ce58640 1 -- 172.21.15.63:0/2555944890 shutdown_connections 2024-03-25T14:29:55.282 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.276+0000 7fca7ce58640 1 -- 172.21.15.63:0/2555944890 wait complete. 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.276+0000 7fca7ce58640 10 monclient: build_initial_monmap 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.276+0000 7fca7ce58640 10 monclient: monmap: 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.277+0000 7fca7ce58640 1 librados: starting msgr at 2024-03-25T14:29:55.283 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.277+0000 7fca7ce58640 1 librados: starting objecter 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.277+0000 7fca7ce58640 1 Processor -- start 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.277+0000 7fca7ce58640 1 -- start start 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.277+0000 7fca7ce58640 1 librados: setting wanted keys 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.277+0000 7fca7ce58640 1 librados: calling monclient init 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.277+0000 7fca7ce58640 10 monclient: init 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 10 monclient: _reopen_session rank -1 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 10 monclient(hunting): picked mon.noname-b con 0x7fca780b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 10 monclient(hunting): picked mon.noname-a con 0x7fca780b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 10 monclient(hunting): picked mon.noname-c con 0x7fca7814be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca781363c0 con 0x7fca780b4d60 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca78137000 con 0x7fca780b1730 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca781372e0 con 0x7fca7814be90 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 10 monclient(hunting): _renew_subs 2024-03-25T14:29:55.284 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.278+0000 7fca7ce58640 10 monclient(hunting): authenticate will time out at 1958.421860s 2024-03-25T14:29:55.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.288+0000 7fca6dd9b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fca780b4d60 0x7fca781dfb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:40434/0 (socket says 172.21.15.63:40434) 2024-03-25T14:29:55.292 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.288+0000 7fca6dd9b640 1 -- 172.21.15.63:0/3837808683 learned_addr learned my addr 172.21.15.63:0/3837808683 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:29:55.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.292+0000 7fca6f7fe640 10 client.?.objecter ms_handle_connect 0x7fca780b4d60 2024-03-25T14:29:55.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.292+0000 7fca6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:55.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.292+0000 7fca6f7fe640 10 client.?.objecter ms_handle_connect 0x7fca780b1730 2024-03-25T14:29:55.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.292+0000 7fca6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:55.299 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.295+0000 7fca6f7fe640 10 client.?.objecter ms_handle_connect 0x7fca7814be90 2024-03-25T14:29:55.299 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.295+0000 7fca6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:29:55.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.305+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855458854 0 0) 0x7fca78137000 con 0x7fca780b1730 2024-03-25T14:29:55.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.305+0000 7fca6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.305+0000 7fca6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.309 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.305+0000 7fca6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.305+0000 7fca6f7fe640 10 monclient(hunting): my global_id is 4684 2024-03-25T14:29:55.310 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.305+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca58004eb0 con 0x7fca780b1730 2024-03-25T14:29:55.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.307+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032111390 0 0) 0x7fca781363c0 con 0x7fca780b4d60 2024-03-25T14:29:55.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.307+0000 7fca6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.307+0000 7fca6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.307+0000 7fca6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.307+0000 7fca6f7fe640 10 monclient(hunting): my global_id is 4680 2024-03-25T14:29:55.311 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.307+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca78137000 con 0x7fca780b4d60 2024-03-25T14:29:55.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.320+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657105602 0 0) 0x7fca781372e0 con 0x7fca7814be90 2024-03-25T14:29:55.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.320+0000 7fca6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:29:55.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.320+0000 7fca6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:29:55.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.320+0000 7fca6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:29:55.324 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.320+0000 7fca6f7fe640 10 monclient(hunting): my global_id is 4655 2024-03-25T14:29:55.325 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.320+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca781363c0 con 0x7fca7814be90 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.327+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2975962873 0 0) 0x7fca58004eb0 con 0x7fca780b1730 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.327+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca781372e0 con 0x7fca780b1730 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.327+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca68001b00 con 0x7fca780b1730 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.327+0000 7fca6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.327+0000 7fca6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.327+0000 7fca6f7fe640 10 monclient(hunting): dump: 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:55.331 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.327+0000 7fca6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.344+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523715238 0 0) 0x7fca781363c0 con 0x7fca7814be90 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.344+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca58004eb0 con 0x7fca7814be90 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.344+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca70001bd0 con 0x7fca7814be90 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.344+0000 7fca6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.344+0000 7fca6f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.344+0000 7fca6f7fe640 10 monclient(hunting): dump: 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:55.348 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:55.349 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:55.349 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:55.349 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.347+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754590435 0 0) 0x7fca78137000 con 0x7fca780b4d60 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.347+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca781363c0 con 0x7fca780b4d60 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.347+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca60001970 con 0x7fca780b4d60 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.347+0000 7fca6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.347+0000 7fca6f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.347+0000 7fca6f7fe640 10 monclient(hunting): dump: 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:55.351 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:55.352 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:55.352 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:55.352 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1632393001 0 0) 0x7fca781372e0 con 0x7fca780b1730 2024-03-25T14:29:55.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 >> v1:172.21.15.63:6790/0 conn(0x7fca7814be90 legacy=0x7fca781e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 >> v1:172.21.15.63:6789/0 conn(0x7fca780b4d60 legacy=0x7fca781dfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.360 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 1 monclient: found mon.b 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7813e1c0 con 0x7fca780b1730 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 5 monclient: authenticate success, global_id 4684 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 10 monclient: _renew_subs 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7813e410 con 0x7fca780b1730 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 10 monclient: _renew_subs 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca78135df0 con 0x7fca780b1730 2024-03-25T14:29:55.361 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.356+0000 7fca7ce58640 1 librados: init done 2024-03-25T14:29:55.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.357+0000 7fca7ce58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:55.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.357+0000 7fca7ce58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:55.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.357+0000 7fca7ce58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:55.362 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.357+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca780627b0 con 0x7fca780b1730 2024-03-25T14:29:55.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.628+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca68001220 con 0x7fca780b1730 2024-03-25T14:29:55.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.628+0000 7fca6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.628+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fca68004c30 con 0x7fca780b1730 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.628+0000 7fca6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.628+0000 7fca6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.628+0000 7fca6f7fe640 10 monclient: dump: 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:29:55.633 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:29:55.634 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:29:55.634 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:29:55.634 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:29:55.634 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:29:55.634 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:29:55.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.628+0000 7fca7559a640 4 set_mon_vals no callback set 2024-03-25T14:29:55.634 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.629+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fca6801ac00 con 0x7fca780b1730 2024-03-25T14:29:55.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.629+0000 7fca6dd9b640 10 monclient: get_auth_request con 0x7fca5806e0f0 auth_method 0 2024-03-25T14:29:55.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.630+0000 7fca6f7fe640 10 client.4684.objecter ms_handle_connect 0x7fca5806e0f0 2024-03-25T14:29:55.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.683+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2678669520 0 0) 0x7fca68054860 con 0x7fca780b1730 2024-03-25T14:29:55.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.683+0000 7fca6f7fe640 10 client.4684.objecter ms_dispatch 0x7fca78000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T14:29:55.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.683+0000 7fca6f7fe640 3 client.4684.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T14:29:55.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.683+0000 7fca6f7fe640 3 client.4684.objecter handle_osd_map decoding full epoch 11 2024-03-25T14:29:55.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.683+0000 7fca6f7fe640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T14:29:55.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.684+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca680501a0 con 0x7fca780b1730 2024-03-25T14:29:55.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.684+0000 7fca6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:29:55.688 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.684+0000 7fca6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:29:55.927 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.923+0000 7fca7ce58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:55.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.923+0000 7fca7ce58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:55.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.923+0000 7fca7ce58640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:29:55.928 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.923+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca78069f40 con 0x7fca780b1730 2024-03-25T14:29:55.973 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.969+0000 7fca6f7fe640 1 -- 172.21.15.63:0/3837808683 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7086 (unknown 238102980 0 1018297132) 0x7fca680522a0 con 0x7fca780b1730 2024-03-25T14:29:55.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.969+0000 7fca6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:29:55.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.969+0000 7fca6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:29:55.974 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:29:55.974 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:52.824245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:52.824245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.972+0000 7fca7ce58640 10 librados: watch_flush enter 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.972+0000 7fca7ce58640 10 librados: watch_flush exit 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.972+0000 7fca7ce58640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.972+0000 7fca7ce58640 10 client.4684.objecter successfully canceled tick 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.972+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 >> v1:172.21.15.153:6800/1688255552 conn(0x7fca5806e0f0 legacy=0x7fca580705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.972+0000 7fca7ce58640 10 monclient: shutdown 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.973+0000 7fca7ce58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:29:55.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.973+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 >> v1:172.21.15.153:6789/0 conn(0x7fca780b1730 legacy=0x7fca781df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:29:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.973+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 shutdown_connections 2024-03-25T14:29:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.973+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 >> 172.21.15.63:0/3837808683 conn(0x7fca780617a0 msgr2=0x7fca780b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:29:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.973+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 shutdown_connections 2024-03-25T14:29:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.973+0000 7fca7ce58640 1 -- 172.21.15.63:0/3837808683 wait complete. 2024-03-25T14:29:55.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:29:55.973+0000 7fca7ce58640 1 librados: shutdown 2024-03-25T14:29:55.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:02.797 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:02.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.905+0000 7f0ce810e640 10 monclient: get_monmap_and_config 2024-03-25T14:30:02.910 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.905+0000 7f0ce810e640 10 monclient: build_initial_monmap 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.905+0000 7f0ce810e640 10 monclient: monmap: 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:02.906923+0000 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:02.906923+0000 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:02.911 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:02.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.906+0000 7f0ce810e640 1 Processor -- start 2024-03-25T14:30:02.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.906+0000 7f0ce810e640 1 -- start start 2024-03-25T14:30:02.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.906+0000 7f0ce810e640 10 monclient: init 2024-03-25T14:30:02.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:02.912 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce00b4d60 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce00b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce014be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00af940 con 0x7f0ce00b1730 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00b0a70 con 0x7f0ce00b4d60 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00b0d50 con 0x7f0ce014be90 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:02.913 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.907+0000 7f0ce810e640 10 monclient(hunting): authenticate will time out at 1966.051864s 2024-03-25T14:30:02.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.912+0000 7f0ce66ab640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0ce014be90 0x7f0ce014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:47110/0 (socket says 172.21.15.63:47110) 2024-03-25T14:30:02.917 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.912+0000 7f0ce66ab640 1 -- 172.21.15.63:0/1126142782 learned_addr learned my addr 172.21.15.63:0/1126142782 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:02.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.928+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623187312 0 0) 0x7f0ce00b0d50 con 0x7f0ce014be90 2024-03-25T14:30:02.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.928+0000 7f0ce4ea8640 10 monclient(hunting): _negotiate 2024-03-25T14:30:02.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.928+0000 7f0ce4ea8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:02.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.928+0000 7f0ce4ea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:02.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.928+0000 7f0ce4ea8640 10 monclient(hunting): my global_id is 4658 2024-03-25T14:30:02.933 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.928+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ccc003520 con 0x7f0ce014be90 2024-03-25T14:30:02.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.940+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460316887 0 0) 0x7f0ce00b0a70 con 0x7f0ce00b4d60 2024-03-25T14:30:02.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.940+0000 7f0ce4ea8640 10 monclient(hunting): _negotiate 2024-03-25T14:30:02.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.940+0000 7f0ce4ea8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:02.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.940+0000 7f0ce4ea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:02.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.940+0000 7f0ce4ea8640 10 monclient(hunting): my global_id is 4687 2024-03-25T14:30:02.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.940+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00b0d50 con 0x7f0ce00b4d60 2024-03-25T14:30:02.947 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.943+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136479441 0 0) 0x7f0ce00af940 con 0x7f0ce00b1730 2024-03-25T14:30:02.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.943+0000 7f0ce4ea8640 10 monclient(hunting): _negotiate 2024-03-25T14:30:02.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.943+0000 7f0ce4ea8640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:02.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.943+0000 7f0ce4ea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:02.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.943+0000 7f0ce4ea8640 10 monclient(hunting): my global_id is 4683 2024-03-25T14:30:02.948 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.943+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce00b0a70 con 0x7f0ce00b1730 2024-03-25T14:30:02.952 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2252234036 0 0) 0x7f0ccc003520 con 0x7f0ce014be90 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 >> v1:172.21.15.153:6789/0 conn(0x7f0ce00b4d60 legacy=0x7f0ce013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 >> v1:172.21.15.63:6789/0 conn(0x7f0ce00b1730 legacy=0x7f0ce00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 10 monclient: _finish_hunting 0 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 1 monclient: found mon.noname-b 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00b1030 con 0x7f0ce014be90 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 10 monclient: _finish_auth 0 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 10 monclient: _check_auth_tickets 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0cd8001a10 con 0x7f0ce014be90 2024-03-25T14:30:02.953 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce810e640 5 monclient: authenticate success, global_id 4658 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce810e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce4ea8640 10 monclient: dump: 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:02.954 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:02.955 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:02.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.947+0000 7f0ce810e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:02.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cd8001b70 con 0x7f0ce014be90 2024-03-25T14:30:02.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce4ea8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:02.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce4ea8640 1 -- 172.21.15.63:0/1126142782 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0cd8001f80 con 0x7f0ce014be90 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce810e640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce810e640 4 set_mon_vals no callback set 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce4ea8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce4ea8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce4ea8640 10 monclient: dump: 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:02.972 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:02.973 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce810e640 10 monclient: shutdown 2024-03-25T14:30:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce810e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.967+0000 7f0ce810e640 1 -- 172.21.15.63:0/1126142782 >> v1:172.21.15.63:6790/0 conn(0x7f0ce014be90 legacy=0x7f0ce014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.968+0000 7f0ce810e640 1 -- 172.21.15.63:0/1126142782 shutdown_connections 2024-03-25T14:30:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.968+0000 7f0ce810e640 1 -- 172.21.15.63:0/1126142782 >> 172.21.15.63:0/1126142782 conn(0x7f0ce00617a0 msgr2=0x7f0ce0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.968+0000 7f0ce810e640 1 -- 172.21.15.63:0/1126142782 shutdown_connections 2024-03-25T14:30:02.974 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.968+0000 7f0ce810e640 1 -- 172.21.15.63:0/1126142782 wait complete. 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 10 monclient: build_initial_monmap 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 10 monclient: monmap: 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 1 librados: starting msgr at 2024-03-25T14:30:02.975 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 1 librados: starting objecter 2024-03-25T14:30:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 1 Processor -- start 2024-03-25T14:30:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 1 -- start start 2024-03-25T14:30:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 1 librados: setting wanted keys 2024-03-25T14:30:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 1 librados: calling monclient init 2024-03-25T14:30:02.976 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.969+0000 7f0ce810e640 10 monclient: init 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce00b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce00b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce014be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce0145360 con 0x7f0ce00b4d60 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce0145840 con 0x7f0ce00b1730 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.970+0000 7f0ce810e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce0137d70 con 0x7f0ce014be90 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.971+0000 7f0ce810e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:02.977 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.971+0000 7f0ce810e640 10 monclient(hunting): authenticate will time out at 1966.115529s 2024-03-25T14:30:02.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.980+0000 7f0ce66ab640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0ce014be90 0x7f0ce012fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:47158/0 (socket says 172.21.15.63:47158) 2024-03-25T14:30:02.984 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.980+0000 7f0ce66ab640 1 -- 172.21.15.63:0/3930481699 learned_addr learned my addr 172.21.15.63:0/3930481699 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:02.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.983+0000 7f0cd67fc640 10 client.?.objecter ms_handle_connect 0x7f0ce014be90 2024-03-25T14:30:02.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.983+0000 7f0cd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:02.994 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.990+0000 7f0cd67fc640 10 client.?.objecter ms_handle_connect 0x7f0ce00b1730 2024-03-25T14:30:02.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.990+0000 7f0cd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:02.995 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.991+0000 7f0cd67fc640 10 client.?.objecter ms_handle_connect 0x7f0ce00b4d60 2024-03-25T14:30:02.996 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.991+0000 7f0cd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:03.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350668269 0 0) 0x7f0ce0145840 con 0x7f0ce00b1730 2024-03-25T14:30:03.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): my global_id is 4690 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc4004eb0 con 0x7f0ce00b1730 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576434624 0 0) 0x7f0ce0137d70 con 0x7f0ce014be90 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 10 monclient(hunting): my global_id is 4661 2024-03-25T14:30:03.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:02.998+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce0145840 con 0x7f0ce014be90 2024-03-25T14:30:03.014 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.009+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820623074 0 0) 0x7f0ce0145360 con 0x7f0ce00b4d60 2024-03-25T14:30:03.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.009+0000 7f0cd67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:03.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.009+0000 7f0cd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:03.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.009+0000 7f0cd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:03.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.009+0000 7f0cd67fc640 10 monclient(hunting): my global_id is 4686 2024-03-25T14:30:03.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.009+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce0137d70 con 0x7f0ce00b4d60 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.011+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874026088 0 0) 0x7f0ce0145840 con 0x7f0ce014be90 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.011+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce0145360 con 0x7f0ce014be90 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.011+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0cd8001e80 con 0x7f0ce014be90 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.011+0000 7f0cd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.011+0000 7f0cd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.011+0000 7f0cd67fc640 10 monclient(hunting): dump: 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:03.016 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:03.017 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:03.017 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:03.017 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:03.017 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.011+0000 7f0cd67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:03.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2886922715 0 0) 0x7f0ce0145360 con 0x7f0ce014be90 2024-03-25T14:30:03.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 >> v1:172.21.15.153:6789/0 conn(0x7f0ce00b1730 legacy=0x7f0ce0135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 >> v1:172.21.15.63:6789/0 conn(0x7f0ce00b4d60 legacy=0x7f0ce01d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:30:03.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 1 monclient: found mon.c 2024-03-25T14:30:03.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce0137f00 con 0x7f0ce014be90 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 10 monclient: _finish_auth 0 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0cd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 5 monclient: authenticate success, global_id 4661 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 10 monclient: _renew_subs 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce01d7eb0 con 0x7f0ce014be90 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 10 monclient: _renew_subs 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce01d8200 con 0x7f0ce014be90 2024-03-25T14:30:03.027 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.021+0000 7f0ce810e640 1 librados: init done 2024-03-25T14:30:03.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.022+0000 7f0ce810e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:03.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.022+0000 7f0ce810e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:03.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.022+0000 7f0ce810e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:30:03.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.022+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce0137aa0 con 0x7f0ce014be90 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.325+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cd8001570 con 0x7f0ce014be90 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.325+0000 7f0cd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.325+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0cd8004e10 con 0x7f0ce014be90 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.325+0000 7f0cd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.325+0000 7f0cd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.325+0000 7f0cd67fc640 10 monclient: dump: 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:03.330 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:03.331 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:03.331 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:03.331 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:03.331 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:03.331 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:03.331 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:03.331 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.325+0000 7f0cd7fff640 4 set_mon_vals no callback set 2024-03-25T14:30:03.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.326+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f0cd801ae60 con 0x7f0ce014be90 2024-03-25T14:30:03.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.326+0000 7f0ce5eaa640 10 monclient: get_auth_request con 0x7f0cc406dd80 auth_method 0 2024-03-25T14:30:03.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.327+0000 7f0cd67fc640 10 client.4661.objecter ms_handle_connect 0x7f0cc406dd80 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7f0cd8054ad0 con 0x7f0ce014be90 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 10 client.4661.objecter ms_dispatch 0x7f0ce0000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 3 client.4661.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 3 client.4661.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cd8050410 con 0x7f0ce014be90 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:03.339 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.334+0000 7f0cd67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:03.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.557+0000 7f0ce810e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:03.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.557+0000 7f0ce810e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:03.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.557+0000 7f0ce810e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:30:03.562 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.557+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ce0130810 con 0x7f0ce014be90 2024-03-25T14:30:03.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.611+0000 7f0cd67fc640 1 -- 172.21.15.63:0/3930481699 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7f0ce0130810 con 0x7f0ce014be90 2024-03-25T14:30:03.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.611+0000 7f0cd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:03.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.611+0000 7f0cd67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:03.617 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:03.617 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:03.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.613+0000 7f0ce810e640 10 librados: watch_flush enter 2024-03-25T14:30:03.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.613+0000 7f0ce810e640 10 librados: watch_flush exit 2024-03-25T14:30:03.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.613+0000 7f0ce810e640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-25T14:30:03.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.613+0000 7f0ce810e640 10 client.4661.objecter successfully canceled tick 2024-03-25T14:30:03.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.613+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 >> v1:172.21.15.153:6800/1688255552 conn(0x7f0cc406dd80 legacy=0x7f0cc4070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 10 monclient: shutdown 2024-03-25T14:30:03.619 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:03.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 >> v1:172.21.15.63:6790/0 conn(0x7f0ce014be90 legacy=0x7f0ce012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:03.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 shutdown_connections 2024-03-25T14:30:03.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 >> 172.21.15.63:0/3930481699 conn(0x7f0ce00617a0 msgr2=0x7f0ce013c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:03.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 shutdown_connections 2024-03-25T14:30:03.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 1 -- 172.21.15.63:0/3930481699 wait complete. 2024-03-25T14:30:03.620 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:03.614+0000 7f0ce810e640 1 librados: shutdown 2024-03-25T14:30:03.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:10.438 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:10.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.549+0000 7fe4ae00d640 10 monclient: get_monmap_and_config 2024-03-25T14:30:10.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.550+0000 7fe4ae00d640 10 monclient: build_initial_monmap 2024-03-25T14:30:10.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.550+0000 7fe4ae00d640 10 monclient: monmap: 2024-03-25T14:30:10.556 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:10.556 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:10.551143+0000 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:10.551143+0000 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:10.557 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.551+0000 7fe4ae00d640 1 Processor -- start 2024-03-25T14:30:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.551+0000 7fe4ae00d640 1 -- start start 2024-03-25T14:30:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.551+0000 7fe4ae00d640 10 monclient: init 2024-03-25T14:30:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:10.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a81430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a8142520 con 0x7fe4a80a98f0 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80b28a0 con 0x7fe4a813f710 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80b2a30 con 0x7fe4a81430c0 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:10.560 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.552+0000 7fe4ae00d640 10 monclient(hunting): authenticate will time out at 1973.696780s 2024-03-25T14:30:10.565 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.560+0000 7fe4a7fff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe4a81430c0 0x7fe4a814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:36306/0 (socket says 172.21.15.63:36306) 2024-03-25T14:30:10.566 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.560+0000 7fe4a7fff640 1 -- 172.21.15.63:0/3696609614 learned_addr learned my addr 172.21.15.63:0/3696609614 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:10.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.584+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397772151 0 0) 0x7fe4a80b28a0 con 0x7fe4a813f710 2024-03-25T14:30:10.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.584+0000 7fe4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.584+0000 7fe4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.584+0000 7fe4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.584+0000 7fe4a67fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T14:30:10.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.584+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe490003520 con 0x7fe4a813f710 2024-03-25T14:30:10.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.592+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764453433 0 0) 0x7fe4a80b2a30 con 0x7fe4a81430c0 2024-03-25T14:30:10.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.592+0000 7fe4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.592+0000 7fe4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.592+0000 7fe4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.592+0000 7fe4a67fc640 10 monclient(hunting): my global_id is 4664 2024-03-25T14:30:10.598 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.592+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a80b28a0 con 0x7fe4a81430c0 2024-03-25T14:30:10.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.593+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569101642 0 0) 0x7fe4a8142520 con 0x7fe4a80a98f0 2024-03-25T14:30:10.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.593+0000 7fe4a67fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.593+0000 7fe4a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.593+0000 7fe4a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.593+0000 7fe4a67fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T14:30:10.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.593+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a80b2a30 con 0x7fe4a80a98f0 2024-03-25T14:30:10.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1154849418 0 0) 0x7fe490003520 con 0x7fe4a813f710 2024-03-25T14:30:10.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 >> v1:172.21.15.63:6790/0 conn(0x7fe4a81430c0 legacy=0x7fe4a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 >> v1:172.21.15.63:6789/0 conn(0x7fe4a80a98f0 legacy=0x7fe4a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 10 monclient: _finish_hunting 0 2024-03-25T14:30:10.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 1 monclient: found mon.noname-c 2024-03-25T14:30:10.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:10.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:30:10.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80b44c0 con 0x7fe4a813f710 2024-03-25T14:30:10.615 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 10 monclient: _finish_auth 0 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 10 monclient: _check_auth_tickets 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe49c001690 con 0x7fe4a813f710 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4ae00d640 5 monclient: authenticate success, global_id 4693 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4ae00d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4a67fc640 10 monclient: dump: 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:10.616 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:10.617 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.608+0000 7fe4ae00d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:10.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.647+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe49c0017f0 con 0x7fe4a813f710 2024-03-25T14:30:10.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.647+0000 7fe4a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:10.654 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4a67fc640 1 -- 172.21.15.63:0/3696609614 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe49c001c00 con 0x7fe4a813f710 2024-03-25T14:30:10.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:10.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 4 set_mon_vals no callback set 2024-03-25T14:30:10.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:10.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:10.655 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4a67fc640 10 monclient: dump: 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:10.656 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 10 monclient: shutdown 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 1 -- 172.21.15.63:0/3696609614 >> v1:172.21.15.153:6789/0 conn(0x7fe4a813f710 legacy=0x7fe4a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 1 -- 172.21.15.63:0/3696609614 shutdown_connections 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 1 -- 172.21.15.63:0/3696609614 >> 172.21.15.63:0/3696609614 conn(0x7fe4a80b7ea0 msgr2=0x7fe4a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.648+0000 7fe4ae00d640 1 -- 172.21.15.63:0/3696609614 shutdown_connections 2024-03-25T14:30:10.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.649+0000 7fe4ae00d640 1 -- 172.21.15.63:0/3696609614 wait complete. 2024-03-25T14:30:10.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.649+0000 7fe4ae00d640 10 monclient: build_initial_monmap 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.649+0000 7fe4ae00d640 10 monclient: monmap: 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:10.660 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 1 librados: starting msgr at 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 1 librados: starting objecter 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 1 Processor -- start 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 1 -- start start 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 1 librados: setting wanted keys 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 1 librados: calling monclient init 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 10 monclient: init 2024-03-25T14:30:10.661 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.650+0000 7fe4ae00d640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a813f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a81430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80abdc0 con 0x7fe4a813f710 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a81d7730 con 0x7fe4a81430c0 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a8138f10 con 0x7fe4a80a98f0 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:10.662 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.651+0000 7fe4ae00d640 10 monclient(hunting): authenticate will time out at 1973.795614s 2024-03-25T14:30:10.670 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.665+0000 7fe4a6ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fe4a813f710 0x7fe4a81d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:41580/0 (socket says 172.21.15.63:41580) 2024-03-25T14:30:10.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.665+0000 7fe4a6ffd640 1 -- 172.21.15.63:0/319784892 learned_addr learned my addr 172.21.15.63:0/319784892 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:10.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.665+0000 7fe48ffff640 10 client.?.objecter ms_handle_connect 0x7fe4a81430c0 2024-03-25T14:30:10.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.665+0000 7fe48ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:10.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.666+0000 7fe48ffff640 10 client.?.objecter ms_handle_connect 0x7fe4a813f710 2024-03-25T14:30:10.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.666+0000 7fe48ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:10.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.671+0000 7fe48ffff640 10 client.?.objecter ms_handle_connect 0x7fe4a80a98f0 2024-03-25T14:30:10.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.671+0000 7fe48ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:10.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.679+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625583872 0 0) 0x7fe4a80abdc0 con 0x7fe4a813f710 2024-03-25T14:30:10.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.679+0000 7fe48ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.679+0000 7fe48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.679+0000 7fe48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.684 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.679+0000 7fe48ffff640 10 monclient(hunting): my global_id is 4692 2024-03-25T14:30:10.685 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.679+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe494004eb0 con 0x7fe4a813f710 2024-03-25T14:30:10.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.685+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542835766 0 0) 0x7fe4a81d7730 con 0x7fe4a81430c0 2024-03-25T14:30:10.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.685+0000 7fe48ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.685+0000 7fe48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.685+0000 7fe48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.685+0000 7fe48ffff640 10 monclient(hunting): my global_id is 4696 2024-03-25T14:30:10.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.685+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a80abdc0 con 0x7fe4a81430c0 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.686+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 172099159 0 0) 0x7fe494004eb0 con 0x7fe4a813f710 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.686+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4a81d7730 con 0x7fe4a813f710 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.686+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe49c001bc0 con 0x7fe4a813f710 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.686+0000 7fe48ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.686+0000 7fe48ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.686+0000 7fe48ffff640 10 monclient(hunting): dump: 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:10.692 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:10.693 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:10.693 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:10.693 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:10.693 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:10.693 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:10.693 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:10.693 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.686+0000 7fe48ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:10.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.690+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547438309 0 0) 0x7fe4a8138f10 con 0x7fe4a80a98f0 2024-03-25T14:30:10.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.690+0000 7fe48ffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:10.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.690+0000 7fe48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:10.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.690+0000 7fe48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:10.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.690+0000 7fe48ffff640 10 monclient(hunting): my global_id is 4667 2024-03-25T14:30:10.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.690+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe494004eb0 con 0x7fe4a80a98f0 2024-03-25T14:30:10.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.699+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1784108930 0 0) 0x7fe4a81d7730 con 0x7fe4a813f710 2024-03-25T14:30:10.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.699+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 >> v1:172.21.15.63:6790/0 conn(0x7fe4a80a98f0 legacy=0x7fe4a8138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.706 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.699+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 >> v1:172.21.15.153:6789/0 conn(0x7fe4a81430c0 legacy=0x7fe4a81d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:10.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 10 monclient: _finish_hunting 0 2024-03-25T14:30:10.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 1 monclient: found mon.a 2024-03-25T14:30:10.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:10.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:10.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a81390a0 con 0x7fe4a813f710 2024-03-25T14:30:10.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 10 monclient: _finish_auth 0 2024-03-25T14:30:10.707 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 10 monclient: _check_auth_tickets 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe48ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 5 monclient: authenticate success, global_id 4692 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 10 monclient: _renew_subs 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a81d8320 con 0x7fe4a813f710 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 10 monclient: _renew_subs 2024-03-25T14:30:10.708 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:10.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a81d43c0 con 0x7fe4a813f710 2024-03-25T14:30:10.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.700+0000 7fe4ae00d640 1 librados: init done 2024-03-25T14:30:10.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.701+0000 7fe4ae00d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:10.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.701+0000 7fe4ae00d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:10.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.701+0000 7fe4ae00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:10.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:10.701+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a80627b0 con 0x7fe4a813f710 2024-03-25T14:30:11.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe49c0011d0 con 0x7fe4a813f710 2024-03-25T14:30:11.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe48ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:11.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe49c003a40 con 0x7fe4a813f710 2024-03-25T14:30:11.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe48ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:11.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe48ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe48ffff640 10 monclient: dump: 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:11.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe4a5ffb640 4 set_mon_vals no callback set 2024-03-25T14:30:11.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.023+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fe49c003c60 con 0x7fe4a813f710 2024-03-25T14:30:11.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.024+0000 7fe4a77fe640 10 monclient: get_auth_request con 0x7fe49406ddb0 auth_method 0 2024-03-25T14:30:11.030 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.024+0000 7fe48ffff640 10 client.4692.objecter ms_handle_connect 0x7fe49406ddb0 2024-03-25T14:30:11.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7fe49c0548c0 con 0x7fe4a813f710 2024-03-25T14:30:11.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 10 client.4692.objecter ms_dispatch 0x7fe4a8000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:11.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 3 client.4692.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:11.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 3 client.4692.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:11.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T14:30:11.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe49c050200 con 0x7fe4a813f710 2024-03-25T14:30:11.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:11.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.032+0000 7fe48ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:11.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.289+0000 7fe4ae00d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:11.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.289+0000 7fe4ae00d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:11.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.289+0000 7fe4ae00d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:11.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.289+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4a8069f40 con 0x7fe4a813f710 2024-03-25T14:30:11.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.348+0000 7fe48ffff640 1 -- 172.21.15.63:0/319784892 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7fe49c052300 con 0x7fe4a813f710 2024-03-25T14:30:11.353 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.348+0000 7fe48ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:11.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.348+0000 7fe48ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:11.354 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:11.354 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 10 librados: watch_flush enter 2024-03-25T14:30:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 10 librados: watch_flush exit 2024-03-25T14:30:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T14:30:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 10 client.4692.objecter successfully canceled tick 2024-03-25T14:30:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 >> v1:172.21.15.153:6800/1688255552 conn(0x7fe49406ddb0 legacy=0x7fe494070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:11.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 10 monclient: shutdown 2024-03-25T14:30:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 >> v1:172.21.15.63:6789/0 conn(0x7fe4a813f710 legacy=0x7fe4a81d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 shutdown_connections 2024-03-25T14:30:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 >> 172.21.15.63:0/319784892 conn(0x7fe4a80b7ea0 msgr2=0x7fe4a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 shutdown_connections 2024-03-25T14:30:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 1 -- 172.21.15.63:0/319784892 wait complete. 2024-03-25T14:30:11.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:11.351+0000 7fe4ae00d640 1 librados: shutdown 2024-03-25T14:30:11.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:13.022 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:51.533 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.033 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:51.544 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.063 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:10:51.574 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.074 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:51.533 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.082 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:51.538 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.101 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:51.613 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.103 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:51.562 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.106 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:51.618 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:13.146 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:10:51.657 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:18.174 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:18.290 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.285+0000 7fd3417e4640 10 monclient: get_monmap_and_config 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.285+0000 7fd3417e4640 10 monclient: build_initial_monmap 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.285+0000 7fd3417e4640 10 monclient: monmap: 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:18.286979+0000 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:18.286979+0000 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:18.291 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:18.292 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:18.292 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.286+0000 7fd3417e4640 1 Processor -- start 2024-03-25T14:30:18.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.286+0000 7fd3417e4640 1 -- start start 2024-03-25T14:30:18.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.286+0000 7fd3417e4640 10 monclient: init 2024-03-25T14:30:18.293 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c0a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c13f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c1430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c142520 con 0x7fd33c1430c0 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b28a0 con 0x7fd33c0a98f0 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b2a30 con 0x7fd33c13f710 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:18.294 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.287+0000 7fd3417e4640 10 monclient(hunting): authenticate will time out at 1981.432328s 2024-03-25T14:30:18.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.291+0000 7fd33affd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd33c0a98f0 0x7fd33c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:38608/0 (socket says 172.21.15.63:38608) 2024-03-25T14:30:18.296 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.291+0000 7fd33affd640 1 -- 172.21.15.63:0/2370035832 learned_addr learned my addr 172.21.15.63:0/2370035832 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:18.306 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.302+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232620430 0 0) 0x7fd33c0b28a0 con 0x7fd33c0a98f0 2024-03-25T14:30:18.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.302+0000 7fd339ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.302+0000 7fd339ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.302+0000 7fd339ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.302+0000 7fd339ffb640 10 monclient(hunting): my global_id is 4699 2024-03-25T14:30:18.307 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.302+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd324003520 con 0x7fd33c0a98f0 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 810959786 0 0) 0x7fd324003520 con 0x7fd33c0a98f0 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 >> v1:172.21.15.63:6790/0 conn(0x7fd33c13f710 legacy=0x7fd33c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 >> v1:172.21.15.63:6789/0 conn(0x7fd33c1430c0 legacy=0x7fd33c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 1 monclient: found mon.noname-c 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0b44c0 con 0x7fd33c0a98f0 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 10 monclient: _finish_auth 0 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:30:18.317 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd32c001690 con 0x7fd33c0a98f0 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd339ffb640 10 monclient: dump: 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:18.318 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd3417e4640 5 monclient: authenticate success, global_id 4699 2024-03-25T14:30:18.319 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.311+0000 7fd3417e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd32c0017f0 con 0x7fd33c0a98f0 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd339ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd339ffb640 1 -- 172.21.15.63:0/2370035832 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd32c001c40 con 0x7fd33c0a98f0 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd339ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd339ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd339ffb640 10 monclient: dump: 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.328 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:18.329 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 4 set_mon_vals no callback set 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 10 monclient: shutdown 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 1 -- 172.21.15.63:0/2370035832 >> v1:172.21.15.153:6789/0 conn(0x7fd33c0a98f0 legacy=0x7fd33c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 1 -- 172.21.15.63:0/2370035832 shutdown_connections 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 1 -- 172.21.15.63:0/2370035832 >> 172.21.15.63:0/2370035832 conn(0x7fd33c0b7ea0 msgr2=0x7fd33c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 1 -- 172.21.15.63:0/2370035832 shutdown_connections 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.323+0000 7fd3417e4640 1 -- 172.21.15.63:0/2370035832 wait complete. 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.324+0000 7fd3417e4640 10 monclient: build_initial_monmap 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.324+0000 7fd3417e4640 10 monclient: monmap: 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:18.332 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.324+0000 7fd3417e4640 1 librados: starting msgr at 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.324+0000 7fd3417e4640 1 librados: starting objecter 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 1 Processor -- start 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 1 -- start start 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 1 librados: setting wanted keys 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 1 librados: calling monclient init 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 10 monclient: init 2024-03-25T14:30:18.333 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c0a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c13f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.325+0000 7fd3417e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c1430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.326+0000 7fd3417e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c138b20 con 0x7fd33c13f710 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.326+0000 7fd3417e4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c137a80 con 0x7fd33c1430c0 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.326+0000 7fd3417e4640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c139880 con 0x7fd33c0a98f0 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.326+0000 7fd3417e4640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:18.334 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.326+0000 7fd3417e4640 10 monclient(hunting): authenticate will time out at 1981.470650s 2024-03-25T14:30:18.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.332+0000 7fd33b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd33c1430c0 0x7fd33c1e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:38646/0 (socket says 172.21.15.63:38646) 2024-03-25T14:30:18.338 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.332+0000 7fd33b7fe640 1 -- 172.21.15.63:0/4127576087 learned_addr learned my addr 172.21.15.63:0/4127576087 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:18.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.338+0000 7fd3237fe640 10 client.?.objecter ms_handle_connect 0x7fd33c0a98f0 2024-03-25T14:30:18.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.338+0000 7fd3237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.340+0000 7fd3237fe640 10 client.?.objecter ms_handle_connect 0x7fd33c1430c0 2024-03-25T14:30:18.345 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.340+0000 7fd3237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:18.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.341+0000 7fd3237fe640 10 client.?.objecter ms_handle_connect 0x7fd33c13f710 2024-03-25T14:30:18.346 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.341+0000 7fd3237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:18.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.350+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472421612 0 0) 0x7fd33c137a80 con 0x7fd33c1430c0 2024-03-25T14:30:18.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.350+0000 7fd3237fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.350+0000 7fd3237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.350+0000 7fd3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.350+0000 7fd3237fe640 10 monclient(hunting): my global_id is 4702 2024-03-25T14:30:18.355 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.350+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd328004eb0 con 0x7fd33c1430c0 2024-03-25T14:30:18.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.351+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104480730 0 0) 0x7fd33c138b20 con 0x7fd33c13f710 2024-03-25T14:30:18.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.351+0000 7fd3237fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.351+0000 7fd3237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.351+0000 7fd3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.351+0000 7fd3237fe640 10 monclient(hunting): my global_id is 4698 2024-03-25T14:30:18.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.351+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c137a80 con 0x7fd33c13f710 2024-03-25T14:30:18.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.358+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238314034 0 0) 0x7fd33c139880 con 0x7fd33c0a98f0 2024-03-25T14:30:18.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.358+0000 7fd3237fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:18.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.358+0000 7fd3237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:18.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.358+0000 7fd3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:18.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.358+0000 7fd3237fe640 10 monclient(hunting): my global_id is 4673 2024-03-25T14:30:18.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.358+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c138b20 con 0x7fd33c0a98f0 2024-03-25T14:30:18.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.367+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 105004837 0 0) 0x7fd33c137a80 con 0x7fd33c13f710 2024-03-25T14:30:18.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.367+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd33c139880 con 0x7fd33c13f710 2024-03-25T14:30:18.372 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.367+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd3300018b0 con 0x7fd33c13f710 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.367+0000 7fd3237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.367+0000 7fd3237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.367+0000 7fd3237fe640 10 monclient(hunting): dump: 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.373 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.367+0000 7fd3237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:18.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.372+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1584469404 0 0) 0x7fd328004eb0 con 0x7fd33c1430c0 2024-03-25T14:30:18.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.372+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd33c137a80 con 0x7fd33c1430c0 2024-03-25T14:30:18.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.372+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd334001c90 con 0x7fd33c1430c0 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.372+0000 7fd3237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.372+0000 7fd3237fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.372+0000 7fd3237fe640 10 monclient(hunting): dump: 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:18.378 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3053618622 0 0) 0x7fd33c138b20 con 0x7fd33c0a98f0 2024-03-25T14:30:18.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd328004eb0 con 0x7fd33c0a98f0 2024-03-25T14:30:18.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd32c001960 con 0x7fd33c0a98f0 2024-03-25T14:30:18.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:18.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 10 monclient(hunting): dump: 2024-03-25T14:30:18.392 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:18.392 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:18.392 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.392 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.392 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.393 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:18.393 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:18.393 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:18.393 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:18.393 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.393 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1300170720 0 0) 0x7fd33c139880 con 0x7fd33c13f710 2024-03-25T14:30:18.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 >> v1:172.21.15.63:6790/0 conn(0x7fd33c0a98f0 legacy=0x7fd33c137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 >> v1:172.21.15.153:6789/0 conn(0x7fd33c1430c0 legacy=0x7fd33c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:18.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 10 monclient: _finish_hunting 0 2024-03-25T14:30:18.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 monclient: found mon.a 2024-03-25T14:30:18.394 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:18.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:18.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c139a10 con 0x7fd33c13f710 2024-03-25T14:30:18.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 10 monclient: _finish_auth 0 2024-03-25T14:30:18.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 10 monclient: _check_auth_tickets 2024-03-25T14:30:18.395 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3417e4640 5 monclient: authenticate success, global_id 4698 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.382+0000 7fd3417e4640 10 monclient: _renew_subs 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.383+0000 7fd3417e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.383+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c1dab00 con 0x7fd33c13f710 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.383+0000 7fd3417e4640 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.383+0000 7fd3417e4640 10 monclient: _renew_subs 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.383+0000 7fd3417e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:18.396 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.383+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c13b4f0 con 0x7fd33c13f710 2024-03-25T14:30:18.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.383+0000 7fd3417e4640 1 librados: init done 2024-03-25T14:30:18.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.384+0000 7fd3417e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:18.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.384+0000 7fd3417e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:18.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.384+0000 7fd3417e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:18.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.384+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c074cc0 con 0x7fd33c13f710 2024-03-25T14:30:18.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.719+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3300035f0 con 0x7fd33c13f710 2024-03-25T14:30:18.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.719+0000 7fd3237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:18.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.719+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd330003a40 con 0x7fd33c13f710 2024-03-25T14:30:18.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.719+0000 7fd3237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:18.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.719+0000 7fd3237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:18.724 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.719+0000 7fd3237fe640 10 monclient: dump: 2024-03-25T14:30:18.724 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:18.725 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.719+0000 7fd3397fa640 4 set_mon_vals no callback set 2024-03-25T14:30:18.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.720+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd330003c60 con 0x7fd33c13f710 2024-03-25T14:30:18.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.721+0000 7fd33affd640 10 monclient: get_auth_request con 0x7fd32806e290 auth_method 0 2024-03-25T14:30:18.726 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.721+0000 7fd3237fe640 10 client.4698.objecter ms_handle_connect 0x7fd32806e290 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.727+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7fd330053570 con 0x7fd33c13f710 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.728+0000 7fd3237fe640 10 client.4698.objecter ms_dispatch 0x7fd33c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.728+0000 7fd3237fe640 3 client.4698.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.728+0000 7fd3237fe640 3 client.4698.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.728+0000 7fd3237fe640 20 client.4698.objecter dump_active .. 0 homeless 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.728+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd33004f080 con 0x7fd33c13f710 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.728+0000 7fd3237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:18.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.728+0000 7fd3237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:18.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.981+0000 7fd3417e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:18.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.981+0000 7fd3417e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:18.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.981+0000 7fd3417e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:30:18.986 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:18.981+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd33c0b3e80 con 0x7fd33c13f710 2024-03-25T14:30:19.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.009+0000 7fd3237fe640 1 -- 172.21.15.63:0/4127576087 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7fd330050fb0 con 0x7fd33c13f710 2024-03-25T14:30:19.019 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.009+0000 7fd3237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:19.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.009+0000 7fd3237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:19.020 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:19.020 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:19.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 10 librados: watch_flush enter 2024-03-25T14:30:19.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 10 librados: watch_flush exit 2024-03-25T14:30:19.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 20 client.4698.objecter shutdown clearing up homeless session... 2024-03-25T14:30:19.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 10 client.4698.objecter successfully canceled tick 2024-03-25T14:30:19.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd32806e290 legacy=0x7fd328070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 10 monclient: shutdown 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.011+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 >> v1:172.21.15.63:6789/0 conn(0x7fd33c13f710 legacy=0x7fd33c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.012+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 shutdown_connections 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.012+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 >> 172.21.15.63:0/4127576087 conn(0x7fd33c0b7ea0 msgr2=0x7fd33c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.012+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 shutdown_connections 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.012+0000 7fd3417e4640 1 -- 172.21.15.63:0/4127576087 wait complete. 2024-03-25T14:30:19.022 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:19.012+0000 7fd3417e4640 1 librados: shutdown 2024-03-25T14:30:19.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:25.836 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:25.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.949+0000 7fab64843640 10 monclient: get_monmap_and_config 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.949+0000 7fab64843640 10 monclient: build_initial_monmap 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.949+0000 7fab64843640 10 monclient: monmap: 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:25.950864+0000 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:25.950864+0000 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:25.955 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.950+0000 7fab64843640 1 Processor -- start 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.950+0000 7fab64843640 1 -- start start 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.950+0000 7fab64843640 10 monclient: init 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 10 monclient(hunting): picked mon.noname-b con 0x7fab600a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 10 monclient(hunting): picked mon.noname-c con 0x7fab6013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 10 monclient(hunting): picked mon.noname-a con 0x7fab601430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab60142520 con 0x7fab601430c0 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab600b28a0 con 0x7fab6013f710 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab600b2a30 con 0x7fab600a98f0 2024-03-25T14:30:25.957 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:25.958 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.951+0000 7fab64843640 10 monclient(hunting): authenticate will time out at 1989.096263s 2024-03-25T14:30:25.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.969+0000 7fab5e59c640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fab600a98f0 0x7fab600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:40964/0 (socket says 172.21.15.63:40964) 2024-03-25T14:30:25.978 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.969+0000 7fab5e59c640 1 -- 172.21.15.63:0/3670112068 learned_addr learned my addr 172.21.15.63:0/3670112068 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:25.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.987+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567967774 0 0) 0x7fab600b2a30 con 0x7fab600a98f0 2024-03-25T14:30:25.992 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.987+0000 7fab5d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:30:25.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.987+0000 7fab5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:25.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.987+0000 7fab5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:25.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.987+0000 7fab5d59a640 10 monclient(hunting): my global_id is 4676 2024-03-25T14:30:25.993 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.987+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab48003520 con 0x7fab600a98f0 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.993+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499017447 0 0) 0x7fab600b28a0 con 0x7fab6013f710 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.993+0000 7fab5d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.993+0000 7fab5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.993+0000 7fab5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.993+0000 7fab5d59a640 10 monclient(hunting): my global_id is 4705 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.993+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab600b2a30 con 0x7fab6013f710 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.994+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961043159 0 0) 0x7fab60142520 con 0x7fab601430c0 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.994+0000 7fab5d59a640 10 monclient(hunting): _negotiate 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.994+0000 7fab5d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.994+0000 7fab5d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.994+0000 7fab5d59a640 10 monclient(hunting): my global_id is 4701 2024-03-25T14:30:25.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.994+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab600b28a0 con 0x7fab601430c0 2024-03-25T14:30:26.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2748408004 0 0) 0x7fab48003520 con 0x7fab600a98f0 2024-03-25T14:30:26.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 >> v1:172.21.15.153:6789/0 conn(0x7fab6013f710 legacy=0x7fab601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 >> v1:172.21.15.63:6789/0 conn(0x7fab601430c0 legacy=0x7fab6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 10 monclient: _finish_hunting 0 2024-03-25T14:30:26.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 1 monclient: found mon.noname-b 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600b44c0 con 0x7fab600a98f0 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 10 monclient: _finish_auth 0 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 10 monclient: _check_auth_tickets 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab64843640 5 monclient: authenticate success, global_id 4676 2024-03-25T14:30:26.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.996+0000 7fab64843640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:26.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.997+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fab50001650 con 0x7fab600a98f0 2024-03-25T14:30:26.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.997+0000 7fab5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.997+0000 7fab5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:26.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.997+0000 7fab5d59a640 10 monclient: dump: 2024-03-25T14:30:26.003 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:26.003 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:26.004 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:25.997+0000 7fab64843640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab500018a0 con 0x7fab600a98f0 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab5d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab5d59a640 1 -- 172.21.15.63:0/3670112068 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fab50001c30 con 0x7fab600a98f0 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab64843640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab64843640 4 set_mon_vals no callback set 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab5d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab5d59a640 10 monclient: dump: 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:26.006 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:26.007 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:26.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab64843640 10 monclient: shutdown 2024-03-25T14:30:26.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab64843640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:26.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.001+0000 7fab64843640 1 -- 172.21.15.63:0/3670112068 >> v1:172.21.15.63:6790/0 conn(0x7fab600a98f0 legacy=0x7fab600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.002+0000 7fab64843640 1 -- 172.21.15.63:0/3670112068 shutdown_connections 2024-03-25T14:30:26.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.002+0000 7fab64843640 1 -- 172.21.15.63:0/3670112068 >> 172.21.15.63:0/3670112068 conn(0x7fab600b7ea0 msgr2=0x7fab600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:26.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.002+0000 7fab64843640 1 -- 172.21.15.63:0/3670112068 shutdown_connections 2024-03-25T14:30:26.009 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.002+0000 7fab64843640 1 -- 172.21.15.63:0/3670112068 wait complete. 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 10 monclient: build_initial_monmap 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 10 monclient: monmap: 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 1 librados: starting msgr at 2024-03-25T14:30:26.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 1 librados: starting objecter 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 1 Processor -- start 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 1 -- start start 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 1 librados: setting wanted keys 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 1 librados: calling monclient init 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.003+0000 7fab64843640 10 monclient: init 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 10 monclient(hunting): picked mon.noname-b con 0x7fab600a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 10 monclient(hunting): picked mon.noname-c con 0x7fab6013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 10 monclient(hunting): picked mon.noname-a con 0x7fab601430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab60136ec0 con 0x7fab601430c0 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab601355f0 con 0x7fab600a98f0 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab601319b0 con 0x7fab6013f710 2024-03-25T14:30:26.011 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:26.012 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.004+0000 7fab64843640 10 monclient(hunting): authenticate will time out at 1989.148921s 2024-03-25T14:30:26.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.013+0000 7fab5e59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fab600a98f0 0x7fab601dac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:51128/0 (socket says 172.21.15.63:51128) 2024-03-25T14:30:26.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.013+0000 7fab5e59c640 1 -- 172.21.15.63:0/690117827 learned_addr learned my addr 172.21.15.63:0/690117827 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:26.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.014+0000 7fab4effd640 10 client.?.objecter ms_handle_connect 0x7fab600a98f0 2024-03-25T14:30:26.018 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.014+0000 7fab4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:26.020 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.016+0000 7fab4effd640 10 client.?.objecter ms_handle_connect 0x7fab601430c0 2024-03-25T14:30:26.021 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.016+0000 7fab4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:26.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 10 client.?.objecter ms_handle_connect 0x7fab6013f710 2024-03-25T14:30:26.028 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:26.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458570779 0 0) 0x7fab601355f0 con 0x7fab600a98f0 2024-03-25T14:30:26.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:26.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:26.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:26.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 10 monclient(hunting): my global_id is 4708 2024-03-25T14:30:26.029 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.024+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab40004eb0 con 0x7fab600a98f0 2024-03-25T14:30:26.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.029+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083341370 0 0) 0x7fab60136ec0 con 0x7fab601430c0 2024-03-25T14:30:26.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.029+0000 7fab4effd640 10 monclient(hunting): _negotiate 2024-03-25T14:30:26.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.029+0000 7fab4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:26.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.029+0000 7fab4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:26.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.029+0000 7fab4effd640 10 monclient(hunting): my global_id is 4704 2024-03-25T14:30:26.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.029+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab601355f0 con 0x7fab601430c0 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.035+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184504095 0 0) 0x7fab40004eb0 con 0x7fab600a98f0 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.035+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab60136ec0 con 0x7fab600a98f0 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.035+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fab50001bb0 con 0x7fab600a98f0 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.035+0000 7fab4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.040 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.035+0000 7fab4effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.035+0000 7fab4effd640 10 monclient(hunting): dump: 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:26.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.035+0000 7fab4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 253738168 0 0) 0x7fab60136ec0 con 0x7fab600a98f0 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 >> v1:172.21.15.63:6790/0 conn(0x7fab6013f710 legacy=0x7fab60130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 >> v1:172.21.15.63:6789/0 conn(0x7fab601430c0 legacy=0x7fab601312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 10 monclient: _finish_hunting 0 2024-03-25T14:30:26.053 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 1 monclient: found mon.b 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60135d80 con 0x7fab600a98f0 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 10 monclient: _finish_auth 0 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 10 monclient: _check_auth_tickets 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 5 monclient: authenticate success, global_id 4708 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 10 monclient: _renew_subs 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab60135f70 con 0x7fab600a98f0 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 10 monclient: _renew_subs 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:26.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6013b780 con 0x7fab600a98f0 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.048+0000 7fab64843640 1 librados: init done 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.049+0000 7fab64843640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.049+0000 7fab64843640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.049+0000 7fab64843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:26.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.049+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab600627b0 con 0x7fab600a98f0 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.330+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab50004bf0 con 0x7fab600a98f0 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.330+0000 7fab4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.330+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fab50004d60 con 0x7fab600a98f0 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.330+0000 7fab4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.330+0000 7fab4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.330+0000 7fab4effd640 10 monclient: dump: 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:26.336 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:26.337 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:26.337 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:26.337 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:26.337 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.330+0000 7fab5cd99640 4 set_mon_vals no callback set 2024-03-25T14:30:26.337 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.332+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fab5001ad30 con 0x7fab600a98f0 2024-03-25T14:30:26.337 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.332+0000 7fab5dd9b640 10 monclient: get_auth_request con 0x7fab4006dd50 auth_method 0 2024-03-25T14:30:26.337 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.332+0000 7fab4effd640 10 client.4708.objecter ms_handle_connect 0x7fab4006dd50 2024-03-25T14:30:26.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.336+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7fab50054840 con 0x7fab600a98f0 2024-03-25T14:30:26.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.336+0000 7fab4effd640 10 client.4708.objecter ms_dispatch 0x7fab60000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:26.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.336+0000 7fab4effd640 3 client.4708.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:26.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.336+0000 7fab4effd640 3 client.4708.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:26.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.336+0000 7fab4effd640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T14:30:26.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.337+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab50050180 con 0x7fab600a98f0 2024-03-25T14:30:26.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.337+0000 7fab4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:26.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.337+0000 7fab4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:26.596 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.591+0000 7fab64843640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.591+0000 7fab64843640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.591+0000 7fab64843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:26.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.591+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab6013e230 con 0x7fab600a98f0 2024-03-25T14:30:26.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.615+0000 7fab4effd640 1 -- 172.21.15.63:0/690117827 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7fab50052280 con 0x7fab600a98f0 2024-03-25T14:30:26.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.615+0000 7fab4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:26.621 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.615+0000 7fab4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:26.621 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:26.621 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 10 librados: watch_flush enter 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 10 librados: watch_flush exit 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 10 client.4708.objecter successfully canceled tick 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 >> v1:172.21.15.153:6800/1688255552 conn(0x7fab4006dd50 legacy=0x7fab40070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 10 monclient: shutdown 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.618+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 >> v1:172.21.15.153:6789/0 conn(0x7fab600a98f0 legacy=0x7fab601dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:26.624 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.619+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 shutdown_connections 2024-03-25T14:30:26.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.619+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 >> 172.21.15.63:0/690117827 conn(0x7fab600b7ea0 msgr2=0x7fab60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:26.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.619+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 shutdown_connections 2024-03-25T14:30:26.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.619+0000 7fab64843640 1 -- 172.21.15.63:0/690117827 wait complete. 2024-03-25T14:30:26.625 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:26.619+0000 7fab64843640 1 librados: shutdown 2024-03-25T14:30:26.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:33.097 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:11.608 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.105 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:11.617 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.127 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:11.639 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.147 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:11.606 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.164 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:11.623 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.192 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:11.704 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.193 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:11.652 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.200 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:11.711 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.224 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:11.736 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:33.442 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:33.553 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.548+0000 7f5b1e1b5640 10 monclient: get_monmap_and_config 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.548+0000 7f5b1e1b5640 10 monclient: build_initial_monmap 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.548+0000 7f5b1e1b5640 10 monclient: monmap: 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:33.550004+0000 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:33.550004+0000 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:33.554 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:33.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.549+0000 7f5b1e1b5640 1 Processor -- start 2024-03-25T14:30:33.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.549+0000 7f5b1e1b5640 1 -- start start 2024-03-25T14:30:33.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.549+0000 7f5b1e1b5640 10 monclient: init 2024-03-25T14:30:33.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:33.555 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 10 monclient(hunting): picked mon.noname-b con 0x7f5b180a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 10 monclient(hunting): picked mon.noname-a con 0x7f5b1813f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 10 monclient(hunting): picked mon.noname-c con 0x7f5b181430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b18142520 con 0x7f5b1813f710 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180b28a0 con 0x7f5b181430c0 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180b2a30 con 0x7f5b180a98f0 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:33.556 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.550+0000 7f5b1e1b5640 10 monclient(hunting): authenticate will time out at 1996.695300s 2024-03-25T14:30:33.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.554+0000 7f5b17fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5b181430c0 0x7f5b1814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36874/0 (socket says 172.21.15.63:36874) 2024-03-25T14:30:33.559 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.554+0000 7f5b17fff640 1 -- 172.21.15.63:0/3520521758 learned_addr learned my addr 172.21.15.63:0/3520521758 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:33.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.565+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454109230 0 0) 0x7f5b180b28a0 con 0x7f5b181430c0 2024-03-25T14:30:33.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.565+0000 7f5b167fc640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.565+0000 7f5b167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.565+0000 7f5b167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.565+0000 7f5b167fc640 10 monclient(hunting): my global_id is 4711 2024-03-25T14:30:33.570 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.565+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b00003520 con 0x7f5b181430c0 2024-03-25T14:30:33.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031784855 0 0) 0x7f5b00003520 con 0x7f5b181430c0 2024-03-25T14:30:33.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 >> v1:172.21.15.63:6790/0 conn(0x7f5b180a98f0 legacy=0x7f5b180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:30:33.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 >> v1:172.21.15.63:6789/0 conn(0x7f5b1813f710 legacy=0x7f5b181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.578 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 10 monclient: _finish_hunting 0 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 1 monclient: found mon.noname-c 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b180b44c0 con 0x7f5b181430c0 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 10 monclient: _finish_auth 0 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 10 monclient: _check_auth_tickets 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5b10001a10 con 0x7f5b181430c0 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b167fc640 10 monclient: dump: 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:33.579 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b1e1b5640 5 monclient: authenticate success, global_id 4711 2024-03-25T14:30:33.580 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.573+0000 7f5b1e1b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:33.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b10001be0 con 0x7f5b181430c0 2024-03-25T14:30:33.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:33.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b167fc640 1 -- 172.21.15.63:0/3520521758 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5b10001ff0 con 0x7f5b181430c0 2024-03-25T14:30:33.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:33.585 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b167fc640 10 monclient: dump: 2024-03-25T14:30:33.585 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b1e1b5640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:33.586 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.580+0000 7f5b1e1b5640 4 set_mon_vals no callback set 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.581+0000 7f5b1e1b5640 10 monclient: shutdown 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.581+0000 7f5b1e1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.581+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/3520521758 >> v1:172.21.15.153:6789/0 conn(0x7f5b181430c0 legacy=0x7f5b1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.581+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/3520521758 shutdown_connections 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.581+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/3520521758 >> 172.21.15.63:0/3520521758 conn(0x7f5b180b7ea0 msgr2=0x7f5b180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.581+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/3520521758 shutdown_connections 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.581+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/3520521758 wait complete. 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.582+0000 7f5b1e1b5640 10 monclient: build_initial_monmap 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.582+0000 7f5b1e1b5640 10 monclient: monmap: 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:33.589 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.582+0000 7f5b1e1b5640 1 librados: starting msgr at 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.582+0000 7f5b1e1b5640 1 librados: starting objecter 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.582+0000 7f5b1e1b5640 1 Processor -- start 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 1 -- start start 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 1 librados: setting wanted keys 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 1 librados: calling monclient init 2024-03-25T14:30:33.590 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient: init 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient(hunting): picked mon.noname-b con 0x7f5b180a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient(hunting): picked mon.noname-a con 0x7f5b1813f710 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient(hunting): picked mon.noname-c con 0x7f5b181430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b181399f0 con 0x7f5b1813f710 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b181386a0 con 0x7f5b180a98f0 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180b1760 con 0x7f5b181430c0 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:33.591 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.583+0000 7f5b1e1b5640 10 monclient(hunting): authenticate will time out at 1996.728403s 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.592+0000 7f5b177fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5b180a98f0 0x7f5b18137830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:36922/0 (socket says 172.21.15.63:36922) 2024-03-25T14:30:33.597 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.592+0000 7f5b177fe640 1 -- 172.21.15.63:0/1582704377 learned_addr learned my addr 172.21.15.63:0/1582704377 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.594+0000 7f5afffff640 10 client.?.objecter ms_handle_connect 0x7f5b180a98f0 2024-03-25T14:30:33.599 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.594+0000 7f5afffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:33.611 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.607+0000 7f5afffff640 10 client.?.objecter ms_handle_connect 0x7f5b181430c0 2024-03-25T14:30:33.612 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.607+0000 7f5afffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765893884 0 0) 0x7f5b181386a0 con 0x7f5b180a98f0 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 10 monclient(hunting): my global_id is 4714 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04004eb0 con 0x7f5b180a98f0 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 10 client.?.objecter ms_handle_connect 0x7f5b1813f710 2024-03-25T14:30:33.614 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.609+0000 7f5afffff640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.623+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512609682 0 0) 0x7f5b04004eb0 con 0x7f5b180a98f0 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.623+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b181386a0 con 0x7f5b180a98f0 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.623+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5b08001960 con 0x7f5b180a98f0 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.623+0000 7f5afffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.623+0000 7f5afffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.623+0000 7f5afffff640 10 monclient(hunting): dump: 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:33.628 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:33.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.623+0000 7f5afffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:33.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.625+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220765735 0 0) 0x7f5b180b1760 con 0x7f5b181430c0 2024-03-25T14:30:33.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.625+0000 7f5afffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.625+0000 7f5afffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.625+0000 7f5afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.625+0000 7f5afffff640 10 monclient(hunting): my global_id is 4682 2024-03-25T14:30:33.630 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.625+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04004eb0 con 0x7f5b181430c0 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.630+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798262244 0 0) 0x7f5b181399f0 con 0x7f5b1813f710 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.630+0000 7f5afffff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.630+0000 7f5afffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.630+0000 7f5afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.630+0000 7f5afffff640 10 monclient(hunting): my global_id is 4710 2024-03-25T14:30:33.635 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.630+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b180b1760 con 0x7f5b1813f710 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.633+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3667053092 0 0) 0x7f5b04004eb0 con 0x7f5b181430c0 2024-03-25T14:30:33.637 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.633+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b181399f0 con 0x7f5b181430c0 2024-03-25T14:30:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2554371788 0 0) 0x7f5b181386a0 con 0x7f5b180a98f0 2024-03-25T14:30:33.642 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 >> v1:172.21.15.63:6790/0 conn(0x7f5b181430c0 legacy=0x7f5b181e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 >> v1:172.21.15.63:6789/0 conn(0x7f5b1813f710 legacy=0x7f5b181da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 10 monclient: _finish_hunting 0 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 1 monclient: found mon.b 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b181380f0 con 0x7f5b180a98f0 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 10 monclient: _finish_auth 0 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 10 monclient: _check_auth_tickets 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5afffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 5 monclient: authenticate success, global_id 4714 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 10 monclient: _renew_subs 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b181383a0 con 0x7f5b180a98f0 2024-03-25T14:30:33.643 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 10 monclient: _renew_subs 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b181db170 con 0x7f5b180a98f0 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.637+0000 7f5b1e1b5640 1 librados: init done 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.638+0000 7f5b1e1b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.638+0000 7f5b1e1b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:33.644 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.638+0000 7f5b1e1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:33.645 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.638+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b181db470 con 0x7f5b180a98f0 2024-03-25T14:30:33.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.909+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b080035f0 con 0x7f5b180a98f0 2024-03-25T14:30:33.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.909+0000 7f5afffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:33.914 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.909+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f5b08003980 con 0x7f5b180a98f0 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.909+0000 7f5afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.909+0000 7f5afffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.909+0000 7f5afffff640 10 monclient: dump: 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:33.915 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:33.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.909+0000 7f5b15ffb640 4 set_mon_vals no callback set 2024-03-25T14:30:33.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.910+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f5b08003ba0 con 0x7f5b180a98f0 2024-03-25T14:30:33.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.911+0000 7f5b16ffd640 10 monclient: get_auth_request con 0x7f5b0406e170 auth_method 0 2024-03-25T14:30:33.916 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.911+0000 7f5afffff640 10 client.4714.objecter ms_handle_connect 0x7f5b0406e170 2024-03-25T14:30:33.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.917+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7f5b080547f0 con 0x7f5b180a98f0 2024-03-25T14:30:33.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.917+0000 7f5afffff640 10 client.4714.objecter ms_dispatch 0x7f5b18000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:33.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.917+0000 7f5afffff640 3 client.4714.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:33.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.917+0000 7f5afffff640 3 client.4714.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:33.922 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.917+0000 7f5afffff640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T14:30:33.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.917+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b08050300 con 0x7f5b180a98f0 2024-03-25T14:30:33.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.918+0000 7f5afffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:33.923 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:33.918+0000 7f5afffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:34.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.156+0000 7f5b1e1b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:34.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.157+0000 7f5b1e1b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:34.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.157+0000 7f5b1e1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:34.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.157+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b1814f320 con 0x7f5b180a98f0 2024-03-25T14:30:34.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.192+0000 7f5afffff640 1 -- 172.21.15.63:0/1582704377 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7f5b08052230 con 0x7f5b180a98f0 2024-03-25T14:30:34.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.192+0000 7f5afffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:34.197 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.192+0000 7f5afffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:34.198 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:34.198 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 10 librados: watch_flush enter 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 10 librados: watch_flush exit 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 10 client.4714.objecter successfully canceled tick 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 >> v1:172.21.15.153:6800/1688255552 conn(0x7f5b0406e170 legacy=0x7f5b04070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 10 monclient: shutdown 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 >> v1:172.21.15.153:6789/0 conn(0x7f5b180a98f0 legacy=0x7f5b18137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 shutdown_connections 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.194+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 >> 172.21.15.63:0/1582704377 conn(0x7f5b180b7ea0 msgr2=0x7f5b18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.195+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 shutdown_connections 2024-03-25T14:30:34.200 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.195+0000 7f5b1e1b5640 1 -- 172.21.15.63:0/1582704377 wait complete. 2024-03-25T14:30:34.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:34.195+0000 7f5b1e1b5640 1 librados: shutdown 2024-03-25T14:30:34.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:41.018 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:41.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.124+0000 7fe6fad23640 10 monclient: get_monmap_and_config 2024-03-25T14:30:41.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.124+0000 7fe6fad23640 10 monclient: build_initial_monmap 2024-03-25T14:30:41.129 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.124+0000 7fe6fad23640 10 monclient: monmap: 2024-03-25T14:30:41.129 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:41.129 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:41.129 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:41.125580+0000 2024-03-25T14:30:41.130 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:41.125580+0000 2024-03-25T14:30:41.130 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:41.130 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:41.130 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:41.130 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:41.130 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:41.130 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:41.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.125+0000 7fe6fad23640 1 Processor -- start 2024-03-25T14:30:41.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.125+0000 7fe6fad23640 1 -- start start 2024-03-25T14:30:41.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.125+0000 7fe6fad23640 10 monclient: init 2024-03-25T14:30:41.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:41.131 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 10 monclient(hunting): picked mon.noname-b con 0x7fe6f40b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 10 monclient(hunting): picked mon.noname-c con 0x7fe6f40b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 10 monclient(hunting): picked mon.noname-a con 0x7fe6f414be90 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f40af940 con 0x7fe6f414be90 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f40b0a70 con 0x7fe6f40b1730 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f40b0d50 con 0x7fe6f40b4d60 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:41.132 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.126+0000 7fe6fad23640 10 monclient(hunting): authenticate will time out at 2004.270728s 2024-03-25T14:30:41.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.137+0000 7fe6f8abf640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7fe6f40b4d60 0x7fe6f413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:49090/0 (socket says 172.21.15.63:49090) 2024-03-25T14:30:41.142 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.137+0000 7fe6f8abf640 1 -- 172.21.15.63:0/1844142568 learned_addr learned my addr 172.21.15.63:0/1844142568 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:41.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.144+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547711247 0 0) 0x7fe6f40b0d50 con 0x7fe6f40b4d60 2024-03-25T14:30:41.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.144+0000 7fe6f37fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:41.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.144+0000 7fe6f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:41.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.144+0000 7fe6f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:41.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.144+0000 7fe6f37fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T14:30:41.149 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.144+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e4003520 con 0x7fe6f40b4d60 2024-03-25T14:30:41.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 783332765 0 0) 0x7fe6e4003520 con 0x7fe6f40b4d60 2024-03-25T14:30:41.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 >> v1:172.21.15.153:6789/0 conn(0x7fe6f40b1730 legacy=0x7fe6f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 >> v1:172.21.15.63:6789/0 conn(0x7fe6f414be90 legacy=0x7fe6f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 10 monclient: _finish_hunting 0 2024-03-25T14:30:41.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 1 monclient: found mon.noname-b 2024-03-25T14:30:41.161 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.63:6790/0 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f40b1030 con 0x7fe6f40b4d60 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 10 monclient: _finish_auth 0 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 10 monclient: _check_auth_tickets 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe6e8001690 con 0x7fe6f40b4d60 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6f37fe640 10 monclient: dump: 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.162 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.163 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:41.163 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:41.163 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:41.163 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:41.163 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:41.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6fad23640 5 monclient: authenticate success, global_id 4685 2024-03-25T14:30:41.163 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.156+0000 7fe6fad23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 <== mon.2 v1:172.21.15.63:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6e80017f0 con 0x7fe6f40b4d60 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6f37fe640 1 -- 172.21.15.63:0/1844142568 <== mon.2 v1:172.21.15.63:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe6e8001c40 con 0x7fe6f40b4d60 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6fad23640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6fad23640 4 set_mon_vals no callback set 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.169+0000 7fe6f37fe640 10 monclient: dump: 2024-03-25T14:30:41.174 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:41.175 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:41.177 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.170+0000 7fe6fad23640 10 monclient: shutdown 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.170+0000 7fe6fad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.170+0000 7fe6fad23640 1 -- 172.21.15.63:0/1844142568 >> v1:172.21.15.63:6790/0 conn(0x7fe6f40b4d60 legacy=0x7fe6f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.170+0000 7fe6fad23640 1 -- 172.21.15.63:0/1844142568 shutdown_connections 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.170+0000 7fe6fad23640 1 -- 172.21.15.63:0/1844142568 >> 172.21.15.63:0/1844142568 conn(0x7fe6f40617a0 msgr2=0x7fe6f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.170+0000 7fe6fad23640 1 -- 172.21.15.63:0/1844142568 shutdown_connections 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.170+0000 7fe6fad23640 1 -- 172.21.15.63:0/1844142568 wait complete. 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 10 monclient: build_initial_monmap 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 10 monclient: monmap: 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:41.178 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 1 librados: starting msgr at 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 1 librados: starting objecter 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 1 Processor -- start 2024-03-25T14:30:41.179 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 1 -- start start 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 1 librados: setting wanted keys 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 1 librados: calling monclient init 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.171+0000 7fe6fad23640 10 monclient: init 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 10 monclient(hunting): picked mon.noname-c con 0x7fe6f40b1730 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 10 monclient(hunting): picked mon.noname-a con 0x7fe6f40b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 10 monclient(hunting): picked mon.noname-b con 0x7fe6f414be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f41d79a0 con 0x7fe6f40b4d60 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f4136ed0 con 0x7fe6f414be90 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f4138280 con 0x7fe6f40b1730 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:41.180 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.172+0000 7fe6fad23640 10 monclient(hunting): authenticate will time out at 2004.317413s 2024-03-25T14:30:41.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.183+0000 7fe6f92c0640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe6f414be90 0x7fe6f41e2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:55288/0 (socket says 172.21.15.63:55288) 2024-03-25T14:30:41.188 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.183+0000 7fe6f92c0640 1 -- 172.21.15.63:0/1623509088 learned_addr learned my addr 172.21.15.63:0/1623509088 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:41.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.184+0000 7fe6f17fa640 10 client.?.objecter ms_handle_connect 0x7fe6f414be90 2024-03-25T14:30:41.189 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.184+0000 7fe6f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:41.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.189+0000 7fe6f17fa640 10 client.?.objecter ms_handle_connect 0x7fe6f40b1730 2024-03-25T14:30:41.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.189+0000 7fe6f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:41.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.189+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551310078 0 0) 0x7fe6f4136ed0 con 0x7fe6f414be90 2024-03-25T14:30:41.194 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.189+0000 7fe6f17fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:41.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.189+0000 7fe6f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:41.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.189+0000 7fe6f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:41.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.189+0000 7fe6f17fa640 10 monclient(hunting): my global_id is 4720 2024-03-25T14:30:41.195 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.190+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d8004eb0 con 0x7fe6f414be90 2024-03-25T14:30:41.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.196+0000 7fe6f17fa640 10 client.?.objecter ms_handle_connect 0x7fe6f40b4d60 2024-03-25T14:30:41.201 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.196+0000 7fe6f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:41.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.199+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3995310032 0 0) 0x7fe6d8004eb0 con 0x7fe6f414be90 2024-03-25T14:30:41.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.199+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6f4136ed0 con 0x7fe6f414be90 2024-03-25T14:30:41.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.199+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe6ec001c90 con 0x7fe6f414be90 2024-03-25T14:30:41.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.199+0000 7fe6f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.199+0000 7fe6f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:41.204 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.199+0000 7fe6f17fa640 10 monclient(hunting): dump: 2024-03-25T14:30:41.204 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:41.205 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.199+0000 7fe6f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.200+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583279067 0 0) 0x7fe6f4138280 con 0x7fe6f40b1730 2024-03-25T14:30:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.200+0000 7fe6f17fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.200+0000 7fe6f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.200+0000 7fe6f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.200+0000 7fe6f17fa640 10 monclient(hunting): my global_id is 4688 2024-03-25T14:30:41.206 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.200+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d8004eb0 con 0x7fe6f40b1730 2024-03-25T14:30:41.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2486436731 0 0) 0x7fe6f4136ed0 con 0x7fe6f414be90 2024-03-25T14:30:41.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 >> v1:172.21.15.63:6790/0 conn(0x7fe6f40b1730 legacy=0x7fe6f41367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 >> v1:172.21.15.63:6789/0 conn(0x7fe6f40b4d60 legacy=0x7fe6f41d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.214 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 10 monclient: _finish_hunting 0 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 1 monclient: found mon.b 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4143900 con 0x7fe6f414be90 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 10 monclient: _finish_auth 0 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 10 monclient: _check_auth_tickets 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6fad23640 5 monclient: authenticate success, global_id 4720 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6fad23640 10 monclient: _renew_subs 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6fad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.208+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f4136000 con 0x7fe6f414be90 2024-03-25T14:30:41.215 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 10 monclient: _renew_subs 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f41d8720 con 0x7fe6f414be90 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 1 librados: init done 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:41.216 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.209+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f41401f0 con 0x7fe6f414be90 2024-03-25T14:30:41.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.487+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6ec003990 con 0x7fe6f414be90 2024-03-25T14:30:41.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.487+0000 7fe6f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:41.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.487+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fe6ec003da0 con 0x7fe6f414be90 2024-03-25T14:30:41.538 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.487+0000 7fe6f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.487+0000 7fe6f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.487+0000 7fe6f17fa640 10 monclient: dump: 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:41.539 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:41.540 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:41.540 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.487+0000 7fe6f2ffd640 4 set_mon_vals no callback set 2024-03-25T14:30:41.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.488+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fe6ec003fc0 con 0x7fe6f414be90 2024-03-25T14:30:41.541 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.488+0000 7fe6f8abf640 10 monclient: get_auth_request con 0x7fe6d806dd90 auth_method 0 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.489+0000 7fe6f17fa640 10 client.4720.objecter ms_handle_connect 0x7fe6d806dd90 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.491+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7fe6ec0549e0 con 0x7fe6f414be90 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.491+0000 7fe6f17fa640 10 client.4720.objecter ms_dispatch 0x7fe6f4000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.491+0000 7fe6f17fa640 3 client.4720.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.491+0000 7fe6f17fa640 3 client.4720.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.491+0000 7fe6f17fa640 20 client.4720.objecter dump_active .. 0 homeless 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.494+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6ec0504f0 con 0x7fe6f414be90 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.494+0000 7fe6f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:41.542 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.494+0000 7fe6f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:41.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.710+0000 7fe6fad23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:41.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.710+0000 7fe6fad23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:41.721 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.710+0000 7fe6fad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:41.722 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.710+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6f41380e0 con 0x7fe6f414be90 2024-03-25T14:30:41.820 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.810+0000 7fe6f17fa640 1 -- 172.21.15.63:0/1623509088 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7fe6f41380e0 con 0x7fe6f414be90 2024-03-25T14:30:41.821 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.810+0000 7fe6f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:41.821 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.810+0000 7fe6f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:41.823 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:41.823 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 10 librados: watch_flush enter 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 10 librados: watch_flush exit 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 20 client.4720.objecter shutdown clearing up homeless session... 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 10 client.4720.objecter successfully canceled tick 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 >> v1:172.21.15.153:6800/1688255552 conn(0x7fe6d806dd90 legacy=0x7fe6d8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 10 monclient: shutdown 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 >> v1:172.21.15.153:6789/0 conn(0x7fe6f414be90 legacy=0x7fe6f41e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:41.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 shutdown_connections 2024-03-25T14:30:41.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 >> 172.21.15.63:0/1623509088 conn(0x7fe6f40617a0 msgr2=0x7fe6f40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:41.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 shutdown_connections 2024-03-25T14:30:41.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 1 -- 172.21.15.63:0/1623509088 wait complete. 2024-03-25T14:30:41.825 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:41.813+0000 7fe6fad23640 1 librados: shutdown 2024-03-25T14:30:41.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:43.957 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:22.469 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.964 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:22.476 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.969 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:22.428 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.972 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:22.431 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.973 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:22.485 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.977 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:22.489 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.992 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:22.504 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.993 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:22.451 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:43.996 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:22.508 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:48.637 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:48.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.742+0000 7f0ad7925640 10 monclient: get_monmap_and_config 2024-03-25T14:30:48.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.742+0000 7f0ad7925640 10 monclient: build_initial_monmap 2024-03-25T14:30:48.747 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.742+0000 7f0ad7925640 10 monclient: monmap: 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:48.743693+0000 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:48.743693+0000 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:48.748 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:48.749 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.743+0000 7f0ad7925640 1 Processor -- start 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.743+0000 7f0ad7925640 1 -- start start 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.743+0000 7f0ad7925640 10 monclient: init 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad00b4d60 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 10 monclient(hunting): picked mon.noname-a con 0x7f0ad00b1730 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 10 monclient(hunting): picked mon.noname-c con 0x7f0ad014be90 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00af940 con 0x7f0ad00b1730 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00b0a70 con 0x7f0ad014be90 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad00b0d50 con 0x7f0ad00b4d60 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:48.750 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.744+0000 7f0ad7925640 10 monclient(hunting): authenticate will time out at 2011.888924s 2024-03-25T14:30:48.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.749+0000 7f0ad56c1640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f0ad00b4d60 0x7f0ad013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:53882/0 (socket says 172.21.15.63:53882) 2024-03-25T14:30:48.754 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.749+0000 7f0ad56c1640 1 -- 172.21.15.63:0/4130632487 learned_addr learned my addr 172.21.15.63:0/4130632487 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:48.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.761+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323568600 0 0) 0x7f0ad00b0a70 con 0x7f0ad014be90 2024-03-25T14:30:48.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.761+0000 7f0ac7fff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.761+0000 7f0ac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.761+0000 7f0ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.761+0000 7f0ac7fff640 10 monclient(hunting): my global_id is 4723 2024-03-25T14:30:48.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.761+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0abc003520 con 0x7f0ad014be90 2024-03-25T14:30:48.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.763+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503613436 0 0) 0x7f0ad00af940 con 0x7f0ad00b1730 2024-03-25T14:30:48.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.763+0000 7f0ac7fff640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.763+0000 7f0ac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.763+0000 7f0ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.763+0000 7f0ac7fff640 10 monclient(hunting): my global_id is 4719 2024-03-25T14:30:48.769 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.763+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad00b0a70 con 0x7f0ad00b1730 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128229295 0 0) 0x7f0abc003520 con 0x7f0ad014be90 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 >> v1:172.21.15.63:6790/0 conn(0x7f0ad00b4d60 legacy=0x7f0ad013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 >> v1:172.21.15.63:6789/0 conn(0x7f0ad00b1730 legacy=0x7f0ad00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 10 monclient: _finish_hunting 0 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 1 monclient: found mon.noname-c 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad00b1030 con 0x7f0ad014be90 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 10 monclient: _finish_auth 0 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 10 monclient: _check_auth_tickets 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:48.773 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0ac8001a10 con 0x7f0ad014be90 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ac7fff640 10 monclient: dump: 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:48.774 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:48.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ad7925640 5 monclient: authenticate success, global_id 4723 2024-03-25T14:30:48.775 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.767+0000 7f0ad7925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ac8001b70 con 0x7f0ad014be90 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ac7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ac7fff640 1 -- 172.21.15.63:0/4130632487 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0ac8001f80 con 0x7f0ad014be90 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ad7925640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ad7925640 4 set_mon_vals no callback set 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ac7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:48.783 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.778+0000 7f0ac7fff640 10 monclient: dump: 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:48.784 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.779+0000 7f0ad7925640 10 monclient: shutdown 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.779+0000 7f0ad7925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.779+0000 7f0ad7925640 1 -- 172.21.15.63:0/4130632487 >> v1:172.21.15.153:6789/0 conn(0x7f0ad014be90 legacy=0x7f0ad014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.779+0000 7f0ad7925640 1 -- 172.21.15.63:0/4130632487 shutdown_connections 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.779+0000 7f0ad7925640 1 -- 172.21.15.63:0/4130632487 >> 172.21.15.63:0/4130632487 conn(0x7f0ad00617a0 msgr2=0x7f0ad0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.779+0000 7f0ad7925640 1 -- 172.21.15.63:0/4130632487 shutdown_connections 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.779+0000 7f0ad7925640 1 -- 172.21.15.63:0/4130632487 wait complete. 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 10 monclient: build_initial_monmap 2024-03-25T14:30:48.787 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 10 monclient: monmap: 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:48.788 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 1 librados: starting msgr at 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 1 librados: starting objecter 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 1 Processor -- start 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 1 -- start start 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 1 librados: setting wanted keys 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 1 librados: calling monclient init 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.780+0000 7f0ad7925640 10 monclient: init 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad00b1730 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 10 monclient(hunting): picked mon.noname-a con 0x7f0ad00b4d60 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 10 monclient(hunting): picked mon.noname-c con 0x7f0ad014be90 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad01302a0 con 0x7f0ad00b4d60 2024-03-25T14:30:48.789 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad0137cc0 con 0x7f0ad00b1730 2024-03-25T14:30:48.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad0137fa0 con 0x7f0ad014be90 2024-03-25T14:30:48.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:48.790 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.781+0000 7f0ad7925640 10 monclient(hunting): authenticate will time out at 2011.926145s 2024-03-25T14:30:48.792 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.788+0000 7f0ad56c1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0ad00b1730 0x7f0ad0143b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:60038/0 (socket says 172.21.15.63:60038) 2024-03-25T14:30:48.793 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.788+0000 7f0ad56c1640 1 -- 172.21.15.63:0/1926825097 learned_addr learned my addr 172.21.15.63:0/1926825097 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:48.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.790+0000 7f0ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f0ad00b1730 2024-03-25T14:30:48.795 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.790+0000 7f0ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.797+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367137791 0 0) 0x7f0ad0137cc0 con 0x7f0ad00b1730 2024-03-25T14:30:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.797+0000 7f0ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.797+0000 7f0ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.797+0000 7f0ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.797+0000 7f0ac5ffb640 10 monclient(hunting): my global_id is 4726 2024-03-25T14:30:48.802 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.797+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab4004eb0 con 0x7f0ad00b1730 2024-03-25T14:30:48.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.810+0000 7f0ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f0ad014be90 2024-03-25T14:30:48.815 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.810+0000 7f0ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:48.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.813+0000 7f0ac5ffb640 10 client.?.objecter ms_handle_connect 0x7f0ad00b4d60 2024-03-25T14:30:48.818 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.813+0000 7f0ac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.817+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976596120 0 0) 0x7f0ab4004eb0 con 0x7f0ad00b1730 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.818+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ad0137cc0 con 0x7f0ad00b1730 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.818+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0acc0018b0 con 0x7f0ad00b1730 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.818+0000 7f0ac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.818+0000 7f0ac5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.818+0000 7f0ac5ffb640 10 monclient(hunting): dump: 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:48.823 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:48.824 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:48.824 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:48.824 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:48.824 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:48.824 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:48.824 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.818+0000 7f0ac5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.823+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998833753 0 0) 0x7f0ad0137fa0 con 0x7f0ad014be90 2024-03-25T14:30:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.823+0000 7f0ac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T14:30:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.823+0000 7f0ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.823+0000 7f0ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.823+0000 7f0ac5ffb640 10 monclient(hunting): my global_id is 4694 2024-03-25T14:30:48.828 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.823+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab4004eb0 con 0x7f0ad014be90 2024-03-25T14:30:48.834 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2185866068 0 0) 0x7f0ad0137cc0 con 0x7f0ad00b1730 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 >> v1:172.21.15.63:6790/0 conn(0x7f0ad014be90 legacy=0x7f0ad01e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 >> v1:172.21.15.63:6789/0 conn(0x7f0ad00b4d60 legacy=0x7f0ad0144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 10 monclient: _finish_hunting 0 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 1 monclient: found mon.b 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0130530 con 0x7f0ad00b1730 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 10 monclient: _finish_auth 0 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 10 monclient: _check_auth_tickets 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 5 monclient: authenticate success, global_id 4726 2024-03-25T14:30:48.835 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 10 monclient: _renew_subs 2024-03-25T14:30:48.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:48.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad0130830 con 0x7f0ad00b1730 2024-03-25T14:30:48.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:48.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 10 monclient: _renew_subs 2024-03-25T14:30:48.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:48.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad0136910 con 0x7f0ad00b1730 2024-03-25T14:30:48.836 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.829+0000 7f0ad7925640 1 librados: init done 2024-03-25T14:30:48.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.831+0000 7f0ad7925640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:48.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.831+0000 7f0ad7925640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:48.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.831+0000 7f0ad7925640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:48.837 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:48.831+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad00b2270 con 0x7f0ad00b1730 2024-03-25T14:30:49.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.111+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0acc0047e0 con 0x7f0ad00b1730 2024-03-25T14:30:49.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.111+0000 7f0ac5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:49.116 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.111+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f0acc004c30 con 0x7f0ad00b1730 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.111+0000 7f0ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.111+0000 7f0ac5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.111+0000 7f0ac5ffb640 10 monclient: dump: 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:49.117 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:49.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.111+0000 7f0ac77fe640 4 set_mon_vals no callback set 2024-03-25T14:30:49.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.112+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f0acc004e50 con 0x7f0ad00b1730 2024-03-25T14:30:49.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.113+0000 7f0ad4ec0640 10 monclient: get_auth_request con 0x7f0ab406de20 auth_method 0 2024-03-25T14:30:49.118 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.113+0000 7f0ac5ffb640 10 client.4726.objecter ms_handle_connect 0x7f0ab406de20 2024-03-25T14:30:49.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.119+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7f0acc0547f0 con 0x7f0ad00b1730 2024-03-25T14:30:49.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.119+0000 7f0ac5ffb640 10 client.4726.objecter ms_dispatch 0x7f0ad0000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:49.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.119+0000 7f0ac5ffb640 3 client.4726.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:49.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.119+0000 7f0ac5ffb640 3 client.4726.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:49.124 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.119+0000 7f0ac5ffb640 20 client.4726.objecter dump_active .. 0 homeless 2024-03-25T14:30:49.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.122+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0acc050300 con 0x7f0ad00b1730 2024-03-25T14:30:49.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.122+0000 7f0ac5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:49.127 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.122+0000 7f0ac5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:49.356 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.351+0000 7f0ad7925640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:49.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.351+0000 7f0ad7925640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:49.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.351+0000 7f0ad7925640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:49.357 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.351+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ad0136c60 con 0x7f0ad00b1730 2024-03-25T14:30:49.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.397+0000 7f0ac5ffb640 1 -- 172.21.15.63:0/1926825097 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7f0acc052230 con 0x7f0ad00b1730 2024-03-25T14:30:49.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.397+0000 7f0ac5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:49.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.397+0000 7f0ac5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:49.403 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:49.404 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:49.406 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 10 librados: watch_flush enter 2024-03-25T14:30:49.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 10 librados: watch_flush exit 2024-03-25T14:30:49.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 20 client.4726.objecter shutdown clearing up homeless session... 2024-03-25T14:30:49.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 10 client.4726.objecter successfully canceled tick 2024-03-25T14:30:49.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 >> v1:172.21.15.153:6800/1688255552 conn(0x7f0ab406de20 legacy=0x7f0ab40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:49.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 10 monclient: shutdown 2024-03-25T14:30:49.407 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:49.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 >> v1:172.21.15.153:6789/0 conn(0x7f0ad00b1730 legacy=0x7f0ad0143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:49.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 shutdown_connections 2024-03-25T14:30:49.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 >> 172.21.15.63:0/1926825097 conn(0x7f0ad00617a0 msgr2=0x7f0ad013c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:49.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 shutdown_connections 2024-03-25T14:30:49.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 1 -- 172.21.15.63:0/1926825097 wait complete. 2024-03-25T14:30:49.408 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:49.400+0000 7f0ad7925640 1 librados: shutdown 2024-03-25T14:30:49.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:30:53.130 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:31.641 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.178 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:31.636 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.185 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:31.697 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.187 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:31.646 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.214 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:31.673 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.229 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:31.741 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.256 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:31.768 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.260 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:31.771 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:53.302 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:31.813 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:30:56.224 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.336+0000 7f64cf23e640 10 monclient: get_monmap_and_config 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.336+0000 7f64cf23e640 10 monclient: build_initial_monmap 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.336+0000 7f64cf23e640 10 monclient: monmap: 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:30:56.337195+0000 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:30:56.337195+0000 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:56.341 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:56.342 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:56.342 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:30:56.342 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:30:56.342 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:56.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.337+0000 7f64cf23e640 1 Processor -- start 2024-03-25T14:30:56.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.337+0000 7f64cf23e640 1 -- start start 2024-03-25T14:30:56.342 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.337+0000 7f64cf23e640 10 monclient: init 2024-03-25T14:30:56.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:56.343 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 10 monclient(hunting): picked mon.noname-a con 0x7f64c80a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 10 monclient(hunting): picked mon.noname-b con 0x7f64c813f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 10 monclient(hunting): picked mon.noname-c con 0x7f64c81430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c8142520 con 0x7f64c80a98f0 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c80b28a0 con 0x7f64c81430c0 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c80b2a30 con 0x7f64c813f710 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:56.344 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.338+0000 7f64cf23e640 10 monclient(hunting): authenticate will time out at 2019.482924s 2024-03-25T14:30:56.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.349+0000 7f64ccfda640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f64c80a98f0 0x7f64c80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36810/0 (socket says 172.21.15.63:36810) 2024-03-25T14:30:56.354 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.350+0000 7f64ccfda640 1 -- 172.21.15.63:0/2082888703 learned_addr learned my addr 172.21.15.63:0/2082888703 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:56.363 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.358+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085100682 0 0) 0x7f64c8142520 con 0x7f64c80a98f0 2024-03-25T14:30:56.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.359+0000 7f64c77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.359+0000 7f64c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.359+0000 7f64c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.359+0000 7f64c77fe640 10 monclient(hunting): my global_id is 4725 2024-03-25T14:30:56.364 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.359+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b0003520 con 0x7f64c80a98f0 2024-03-25T14:30:56.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.364+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588947929 0 0) 0x7f64c80b28a0 con 0x7f64c81430c0 2024-03-25T14:30:56.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.364+0000 7f64c77fe640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.364+0000 7f64c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.364+0000 7f64c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.369 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.364+0000 7f64c77fe640 10 monclient(hunting): my global_id is 4729 2024-03-25T14:30:56.370 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.364+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c8142520 con 0x7f64c81430c0 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.369+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153984082 0 0) 0x7f64b0003520 con 0x7f64c80a98f0 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.369+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 >> v1:172.21.15.63:6790/0 conn(0x7f64c813f710 legacy=0x7f64c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.369+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 >> v1:172.21.15.153:6789/0 conn(0x7f64c81430c0 legacy=0x7f64c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 10 monclient: _finish_hunting 0 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 1 monclient: found mon.noname-a 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c80b44c0 con 0x7f64c80a98f0 2024-03-25T14:30:56.375 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 10 monclient: _finish_auth 0 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 10 monclient: _check_auth_tickets 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f64b8001690 con 0x7f64c80a98f0 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64c77fe640 10 monclient: dump: 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:56.376 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:56.377 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:56.377 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:56.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64cf23e640 5 monclient: authenticate success, global_id 4725 2024-03-25T14:30:56.377 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.370+0000 7f64cf23e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:30:56.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b80017f0 con 0x7f64c80a98f0 2024-03-25T14:30:56.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:56.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64c77fe640 1 -- 172.21.15.63:0/2082888703 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f64b8001c00 con 0x7f64c80a98f0 2024-03-25T14:30:56.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:30:56.385 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64c77fe640 10 monclient: dump: 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:56.386 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:56.387 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:56.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64cf23e640 10 monclient: get_monmap_and_config success 2024-03-25T14:30:56.387 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.380+0000 7f64cf23e640 4 set_mon_vals no callback set 2024-03-25T14:30:56.388 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.381+0000 7f64cf23e640 10 monclient: shutdown 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.381+0000 7f64cf23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.381+0000 7f64cf23e640 1 -- 172.21.15.63:0/2082888703 >> v1:172.21.15.63:6789/0 conn(0x7f64c80a98f0 legacy=0x7f64c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.381+0000 7f64cf23e640 1 -- 172.21.15.63:0/2082888703 shutdown_connections 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.381+0000 7f64cf23e640 1 -- 172.21.15.63:0/2082888703 >> 172.21.15.63:0/2082888703 conn(0x7f64c80b7ea0 msgr2=0x7f64c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.381+0000 7f64cf23e640 1 -- 172.21.15.63:0/2082888703 shutdown_connections 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.381+0000 7f64cf23e640 1 -- 172.21.15.63:0/2082888703 wait complete. 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 10 monclient: build_initial_monmap 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 10 monclient: monmap: 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:30:56.389 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 1 librados: starting msgr at 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 1 librados: starting objecter 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 1 Processor -- start 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 1 -- start start 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 1 librados: setting wanted keys 2024-03-25T14:30:56.390 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 1 librados: calling monclient init 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.382+0000 7f64cf23e640 10 monclient: init 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 10 monclient: _reopen_session rank -1 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 10 monclient(hunting): picked mon.noname-c con 0x7f64c80a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 10 monclient(hunting): picked mon.noname-b con 0x7f64c813f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 10 monclient(hunting): picked mon.noname-a con 0x7f64c81430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c813ad50 con 0x7f64c81430c0 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c8138e70 con 0x7f64c813f710 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c813af80 con 0x7f64c80a98f0 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 10 monclient(hunting): _renew_subs 2024-03-25T14:30:56.391 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.383+0000 7f64cf23e640 10 monclient(hunting): authenticate will time out at 2019.528063s 2024-03-25T14:30:56.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.387+0000 7f64c7fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f64c813f710 0x7f64c8139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:42756/0 (socket says 172.21.15.63:42756) 2024-03-25T14:30:56.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.387+0000 7f64c7fff640 1 -- 172.21.15.63:0/4183803593 learned_addr learned my addr 172.21.15.63:0/4183803593 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:30:56.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.387+0000 7f64c57fa640 10 client.?.objecter ms_handle_connect 0x7f64c813f710 2024-03-25T14:30:56.392 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.388+0000 7f64c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:56.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.396+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168532840 0 0) 0x7f64c8138e70 con 0x7f64c813f710 2024-03-25T14:30:56.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.396+0000 7f64c57fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.396+0000 7f64c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.396+0000 7f64c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.396+0000 7f64c57fa640 10 monclient(hunting): my global_id is 4732 2024-03-25T14:30:56.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.396+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b4004eb0 con 0x7f64c813f710 2024-03-25T14:30:56.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.399+0000 7f64c57fa640 10 client.?.objecter ms_handle_connect 0x7f64c80a98f0 2024-03-25T14:30:56.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.399+0000 7f64c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:56.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.406+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 766763902 0 0) 0x7f64b4004eb0 con 0x7f64c813f710 2024-03-25T14:30:56.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.406+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64c8138e70 con 0x7f64c813f710 2024-03-25T14:30:56.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.406+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f64c0004400 con 0x7f64c813f710 2024-03-25T14:30:56.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.406+0000 7f64c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.407+0000 7f64c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.407+0000 7f64c57fa640 10 monclient(hunting): dump: 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:56.413 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:56.414 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:56.414 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:56.414 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:56.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.407+0000 7f64c57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:30:56.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.407+0000 7f64c57fa640 10 client.?.objecter ms_handle_connect 0x7f64c81430c0 2024-03-25T14:30:56.414 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.407+0000 7f64c57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:30:56.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.411+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915292158 0 0) 0x7f64c813af80 con 0x7f64c80a98f0 2024-03-25T14:30:56.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.411+0000 7f64c57fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.416 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.411+0000 7f64c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.411+0000 7f64c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.411+0000 7f64c57fa640 10 monclient(hunting): my global_id is 4697 2024-03-25T14:30:56.417 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.411+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b4004eb0 con 0x7f64c80a98f0 2024-03-25T14:30:56.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.415+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917709764 0 0) 0x7f64c813ad50 con 0x7f64c81430c0 2024-03-25T14:30:56.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.415+0000 7f64c57fa640 10 monclient(hunting): _negotiate 2024-03-25T14:30:56.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.415+0000 7f64c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:30:56.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.415+0000 7f64c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:30:56.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.415+0000 7f64c57fa640 10 monclient(hunting): my global_id is 4728 2024-03-25T14:30:56.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.415+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c813af80 con 0x7f64c81430c0 2024-03-25T14:30:56.422 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1012836044 0 0) 0x7f64c8138e70 con 0x7f64c813f710 2024-03-25T14:30:56.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 >> v1:172.21.15.63:6790/0 conn(0x7f64c80a98f0 legacy=0x7f64c81412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 >> v1:172.21.15.63:6789/0 conn(0x7f64c81430c0 legacy=0x7f64c81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 10 monclient: _finish_hunting 0 2024-03-25T14:30:56.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 1 monclient: found mon.b 2024-03-25T14:30:56.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:30:56.423 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c813b260 con 0x7f64c813f710 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 10 monclient: _finish_auth 0 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 10 monclient: _check_auth_tickets 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 5 monclient: authenticate success, global_id 4732 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 10 monclient: _renew_subs 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c813b540 con 0x7f64c813f710 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 10 monclient: _renew_subs 2024-03-25T14:30:56.424 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:56.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c813a430 con 0x7f64c813f710 2024-03-25T14:30:56.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.417+0000 7f64cf23e640 1 librados: init done 2024-03-25T14:30:56.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.418+0000 7f64cf23e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:56.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.418+0000 7f64cf23e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:56.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.418+0000 7f64cf23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:56.425 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.418+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c80627b0 con 0x7f64c813f710 2024-03-25T14:30:56.701 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.695+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64c0001250 con 0x7f64c813f710 2024-03-25T14:30:56.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.695+0000 7f64c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:30:56.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.695+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f64c0001620 con 0x7f64c813f710 2024-03-25T14:30:56.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.695+0000 7f64c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:30:56.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.695+0000 7f64c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:30:56.702 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.695+0000 7f64c57fa640 10 monclient: dump: 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:30:56.703 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.695+0000 7f64c6ffd640 4 set_mon_vals no callback set 2024-03-25T14:30:56.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.696+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f64c0001840 con 0x7f64c813f710 2024-03-25T14:30:56.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.697+0000 7f64ccfda640 10 monclient: get_auth_request con 0x7f64b406de10 auth_method 0 2024-03-25T14:30:56.704 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.697+0000 7f64c57fa640 10 client.4732.objecter ms_handle_connect 0x7f64b406de10 2024-03-25T14:30:56.709 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.704+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7f64c0054ba0 con 0x7f64c813f710 2024-03-25T14:30:56.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.704+0000 7f64c57fa640 10 client.4732.objecter ms_dispatch 0x7f64c8000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:30:56.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.704+0000 7f64c57fa640 3 client.4732.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:30:56.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.704+0000 7f64c57fa640 3 client.4732.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:30:56.710 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.704+0000 7f64c57fa640 20 client.4732.objecter dump_active .. 0 homeless 2024-03-25T14:30:56.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.705+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64c00506b0 con 0x7f64c813f710 2024-03-25T14:30:56.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.705+0000 7f64c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:30:56.711 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.705+0000 7f64c57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:30:56.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.957+0000 7f64cf23e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:56.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.957+0000 7f64cf23e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:56.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.957+0000 7f64cf23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:30:56.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.957+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64c813d780 con 0x7f64c813f710 2024-03-25T14:30:56.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.983+0000 7f64c57fa640 1 -- 172.21.15.63:0/4183803593 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7f64c00525e0 con 0x7f64c813f710 2024-03-25T14:30:56.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.983+0000 7f64c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:30:56.988 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.983+0000 7f64c57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:30:56.988 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:30:56.988 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:30:56.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.984+0000 7f64cf23e640 10 librados: watch_flush enter 2024-03-25T14:30:56.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 10 librados: watch_flush exit 2024-03-25T14:30:56.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 20 client.4732.objecter shutdown clearing up homeless session... 2024-03-25T14:30:56.990 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 10 client.4732.objecter successfully canceled tick 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 >> v1:172.21.15.153:6800/1688255552 conn(0x7f64b406de10 legacy=0x7f64b40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 10 monclient: shutdown 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 >> v1:172.21.15.153:6789/0 conn(0x7f64c813f710 legacy=0x7f64c8139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 shutdown_connections 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 >> 172.21.15.63:0/4183803593 conn(0x7f64c80b7ea0 msgr2=0x7f64c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 shutdown_connections 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 1 -- 172.21.15.63:0/4183803593 wait complete. 2024-03-25T14:30:56.991 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:30:56.985+0000 7f64cf23e640 1 librados: shutdown 2024-03-25T14:30:57.001 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:03.809 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.923+0000 7fa586eed640 10 monclient: get_monmap_and_config 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.923+0000 7fa586eed640 10 monclient: build_initial_monmap 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.923+0000 7fa586eed640 10 monclient: monmap: 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:31:03.925032+0000 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:31:03.925032+0000 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:31:03.929 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:31:03.930 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:31:03.930 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:03.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.925+0000 7fa586eed640 1 Processor -- start 2024-03-25T14:31:03.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.925+0000 7fa586eed640 1 -- start start 2024-03-25T14:31:03.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.925+0000 7fa586eed640 10 monclient: init 2024-03-25T14:31:03.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.925+0000 7fa586eed640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:03.931 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.925+0000 7fa586eed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 10 monclient(hunting): picked mon.noname-b con 0x7fa5800a98f0 addr v1:172.21.15.63:6790/0 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 10 monclient(hunting): picked mon.noname-c con 0x7fa58013f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 10 monclient(hunting): picked mon.noname-a con 0x7fa5801430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa580142520 con 0x7fa5801430c0 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5800b28a0 con 0x7fa58013f710 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5800b2a30 con 0x7fa5800a98f0 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:03.932 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.926+0000 7fa586eed640 10 monclient(hunting): authenticate will time out at 2027.070731s 2024-03-25T14:31:03.945 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.940+0000 7fa57ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa58013f710 0x7fa5801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:37276/0 (socket says 172.21.15.63:37276) 2024-03-25T14:31:03.946 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.940+0000 7fa57ffff640 1 -- 172.21.15.63:0/1907875290 learned_addr learned my addr 172.21.15.63:0/1907875290 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:31:03.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.949+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045463292 0 0) 0x7fa580142520 con 0x7fa5801430c0 2024-03-25T14:31:03.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.949+0000 7fa57f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:03.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.949+0000 7fa57f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:03.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.949+0000 7fa57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:03.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.949+0000 7fa57f7fe640 10 monclient(hunting): my global_id is 4731 2024-03-25T14:31:03.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.949+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570003520 con 0x7fa5801430c0 2024-03-25T14:31:03.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.958+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961077695 0 0) 0x7fa5800b2a30 con 0x7fa5800a98f0 2024-03-25T14:31:03.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.958+0000 7fa57f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:03.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.958+0000 7fa57f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:03.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.958+0000 7fa57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:03.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.958+0000 7fa57f7fe640 10 monclient(hunting): my global_id is 4700 2024-03-25T14:31:03.963 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.958+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa580142520 con 0x7fa5800a98f0 2024-03-25T14:31:03.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.961+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883532795 0 0) 0x7fa5800b28a0 con 0x7fa58013f710 2024-03-25T14:31:03.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.961+0000 7fa57f7fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:03.966 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.961+0000 7fa57f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:03.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.961+0000 7fa57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:03.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.961+0000 7fa57f7fe640 10 monclient(hunting): my global_id is 4735 2024-03-25T14:31:03.967 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.961+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5800b2a30 con 0x7fa58013f710 2024-03-25T14:31:03.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667780077 0 0) 0x7fa570003520 con 0x7fa5801430c0 2024-03-25T14:31:03.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 >> v1:172.21.15.63:6790/0 conn(0x7fa5800a98f0 legacy=0x7fa5800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:03.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 >> v1:172.21.15.153:6789/0 conn(0x7fa58013f710 legacy=0x7fa5801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:03.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 10 monclient: _finish_hunting 0 2024-03-25T14:31:03.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 1 monclient: found mon.noname-a 2024-03-25T14:31:03.969 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.63:6789/0 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5800b44c0 con 0x7fa5801430c0 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 10 monclient: _finish_auth 0 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 10 monclient: _check_auth_tickets 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa578001a10 con 0x7fa5801430c0 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa57f7fe640 10 monclient: dump: 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:03.970 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:03.971 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:03.971 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:03.971 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:03.971 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:03.971 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:03.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa586eed640 5 monclient: authenticate success, global_id 4731 2024-03-25T14:31:03.971 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.964+0000 7fa586eed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:03.997 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 <== mon.0 v1:172.21.15.63:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa578001bf0 con 0x7fa5801430c0 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa57f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa57f7fe640 1 -- 172.21.15.63:0/1907875290 <== mon.0 v1:172.21.15.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa578002000 con 0x7fa5801430c0 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa57f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa57f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa57f7fe640 10 monclient: dump: 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:03.998 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa586eed640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:03.999 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.992+0000 7fa586eed640 4 set_mon_vals no callback set 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.993+0000 7fa586eed640 10 monclient: shutdown 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.993+0000 7fa586eed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.993+0000 7fa586eed640 1 -- 172.21.15.63:0/1907875290 >> v1:172.21.15.63:6789/0 conn(0x7fa5801430c0 legacy=0x7fa58014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.993+0000 7fa586eed640 1 -- 172.21.15.63:0/1907875290 shutdown_connections 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.993+0000 7fa586eed640 1 -- 172.21.15.63:0/1907875290 >> 172.21.15.63:0/1907875290 conn(0x7fa5800b7ea0 msgr2=0x7fa5800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.993+0000 7fa586eed640 1 -- 172.21.15.63:0/1907875290 shutdown_connections 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.993+0000 7fa586eed640 1 -- 172.21.15.63:0/1907875290 wait complete. 2024-03-25T14:31:04.001 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 10 monclient: build_initial_monmap 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 10 monclient: monmap: 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 1 librados: starting msgr at 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 1 librados: starting objecter 2024-03-25T14:31:04.002 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 1 Processor -- start 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 1 -- start start 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 1 librados: setting wanted keys 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 1 librados: calling monclient init 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.994+0000 7fa586eed640 10 monclient: init 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 10 monclient(hunting): picked mon.noname-b con 0x7fa5800a98f0 addr v1:172.21.15.153:6789/0 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 10 monclient(hunting): picked mon.noname-c con 0x7fa58013f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 10 monclient(hunting): picked mon.noname-a con 0x7fa5801430c0 addr v1:172.21.15.63:6789/0 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa580136ed0 con 0x7fa5801430c0 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa580136680 con 0x7fa5800a98f0 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5801d7380 con 0x7fa58013f710 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:04.003 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:03.995+0000 7fa586eed640 10 monclient(hunting): authenticate will time out at 2027.140190s 2024-03-25T14:31:04.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.005+0000 7fa584c89640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa5800a98f0 0x7fa5800b0ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:37322/0 (socket says 172.21.15.63:37322) 2024-03-25T14:31:04.010 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.005+0000 7fa584c89640 1 -- 172.21.15.63:0/1925721419 learned_addr learned my addr 172.21.15.63:0/1925721419 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:31:04.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.010+0000 7fa57d7fa640 10 client.?.objecter ms_handle_connect 0x7fa58013f710 2024-03-25T14:31:04.015 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.010+0000 7fa57d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:04.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.011+0000 7fa57d7fa640 10 client.?.objecter ms_handle_connect 0x7fa5800a98f0 2024-03-25T14:31:04.016 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.011+0000 7fa57d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:04.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.019+0000 7fa57d7fa640 10 client.?.objecter ms_handle_connect 0x7fa5801430c0 2024-03-25T14:31:04.024 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.019+0000 7fa57d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:04.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.027+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053547143 0 0) 0x7fa580136680 con 0x7fa5800a98f0 2024-03-25T14:31:04.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.027+0000 7fa57d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.027+0000 7fa57d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.032 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.027+0000 7fa57d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.027+0000 7fa57d7fa640 10 monclient(hunting): my global_id is 4738 2024-03-25T14:31:04.033 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.027+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa564004eb0 con 0x7fa5800a98f0 2024-03-25T14:31:04.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.029+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383311244 0 0) 0x7fa580136ed0 con 0x7fa5801430c0 2024-03-25T14:31:04.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.029+0000 7fa57d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.029+0000 7fa57d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.034 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.029+0000 7fa57d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.029+0000 7fa57d7fa640 10 monclient(hunting): my global_id is 4734 2024-03-25T14:31:04.035 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.029+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa580136680 con 0x7fa5801430c0 2024-03-25T14:31:04.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.036+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235432268 0 0) 0x7fa5801d7380 con 0x7fa58013f710 2024-03-25T14:31:04.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.036+0000 7fa57d7fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:04.041 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.036+0000 7fa57d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:04.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.036+0000 7fa57d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:04.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.036+0000 7fa57d7fa640 10 monclient(hunting): my global_id is 4703 2024-03-25T14:31:04.042 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.036+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa580136ed0 con 0x7fa58013f710 2024-03-25T14:31:04.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.041+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563621134 0 0) 0x7fa580136680 con 0x7fa5801430c0 2024-03-25T14:31:04.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.041+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5801d7380 con 0x7fa5801430c0 2024-03-25T14:31:04.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.041+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa578001f00 con 0x7fa5801430c0 2024-03-25T14:31:04.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.041+0000 7fa57d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.041+0000 7fa57d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.041+0000 7fa57d7fa640 10 monclient(hunting): dump: 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:04.047 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:04.048 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:04.048 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:04.048 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:04.048 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.041+0000 7fa57d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1222184645 0 0) 0x7fa580136ed0 con 0x7fa58013f710 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa580136680 con 0x7fa58013f710 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa56c001830 con 0x7fa58013f710 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 10 monclient(hunting): dump: 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:04.056 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 468597637 0 0) 0x7fa564004eb0 con 0x7fa5800a98f0 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa580136ed0 con 0x7fa5800a98f0 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa574001930 con 0x7fa5800a98f0 2024-03-25T14:31:04.057 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.051+0000 7fa57d7fa640 10 monclient(hunting): dump: 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:04.058 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:04.071 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1070366183 0 0) 0x7fa5801d7380 con 0x7fa5801430c0 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 >> v1:172.21.15.63:6790/0 conn(0x7fa58013f710 legacy=0x7fa580137340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 >> v1:172.21.15.153:6789/0 conn(0x7fa5800a98f0 legacy=0x7fa5800b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 10 monclient: _finish_hunting 0 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 1 monclient: found mon.a 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5801d7510 con 0x7fa5801430c0 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 10 monclient: _finish_auth 0 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 10 monclient: _check_auth_tickets 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa57d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 5 monclient: authenticate success, global_id 4734 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 10 monclient: _renew_subs 2024-03-25T14:31:04.072 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:31:04.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5800b12b0 con 0x7fa5801430c0 2024-03-25T14:31:04.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:04.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 10 monclient: _renew_subs 2024-03-25T14:31:04.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:31:04.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa580137c30 con 0x7fa5801430c0 2024-03-25T14:31:04.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.066+0000 7fa586eed640 1 librados: init done 2024-03-25T14:31:04.073 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.067+0000 7fa586eed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:04.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.067+0000 7fa586eed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:04.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.067+0000 7fa586eed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:31:04.074 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.067+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5800627b0 con 0x7fa5801430c0 2024-03-25T14:31:04.402 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.397+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5780020e0 con 0x7fa5801430c0 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.397+0000 7fa57d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.397+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fa578003dc0 con 0x7fa5801430c0 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.397+0000 7fa57d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.397+0000 7fa57d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.397+0000 7fa57d7fa640 10 monclient: dump: 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:04.403 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:04.404 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:04.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.397+0000 7fa57effd640 4 set_mon_vals no callback set 2024-03-25T14:31:04.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.398+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fa578019d90 con 0x7fa5801430c0 2024-03-25T14:31:04.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.399+0000 7fa584c89640 10 monclient: get_auth_request con 0x7fa56406e0f0 auth_method 0 2024-03-25T14:31:04.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.399+0000 7fa57d7fa640 10 client.4734.objecter ms_handle_connect 0x7fa56406e0f0 2024-03-25T14:31:04.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7fa578053970 con 0x7fa5801430c0 2024-03-25T14:31:04.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 10 client.4734.objecter ms_dispatch 0x7fa580000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:31:04.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 3 client.4734.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:31:04.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 3 client.4734.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:31:04.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 20 client.4734.objecter dump_active .. 0 homeless 2024-03-25T14:31:04.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa57804f2b0 con 0x7fa5801430c0 2024-03-25T14:31:04.410 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:04.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.405+0000 7fa57d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:04.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.653+0000 7fa586eed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:04.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.654+0000 7fa586eed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:04.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.654+0000 7fa586eed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.63:6789/0 2024-03-25T14:31:04.659 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.654+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 --> v1:172.21.15.63:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5800b1440 con 0x7fa5801430c0 2024-03-25T14:31:04.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.682+0000 7fa57d7fa640 1 -- 172.21.15.63:0/1925721419 <== mon.0 v1:172.21.15.63:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7fa5780513b0 con 0x7fa5801430c0 2024-03-25T14:31:04.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.682+0000 7fa57d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:04.687 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.682+0000 7fa57d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:04.688 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:31:04.688 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:04.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 10 librados: watch_flush enter 2024-03-25T14:31:04.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 10 librados: watch_flush exit 2024-03-25T14:31:04.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 20 client.4734.objecter shutdown clearing up homeless session... 2024-03-25T14:31:04.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 10 client.4734.objecter successfully canceled tick 2024-03-25T14:31:04.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 >> v1:172.21.15.153:6800/1688255552 conn(0x7fa56406e0f0 legacy=0x7fa5640705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 10 monclient: shutdown 2024-03-25T14:31:04.690 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:04.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.684+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 >> v1:172.21.15.63:6789/0 conn(0x7fa5801430c0 legacy=0x7fa5801e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:04.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.685+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 shutdown_connections 2024-03-25T14:31:04.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.685+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 >> 172.21.15.63:0/1925721419 conn(0x7fa5800b7ea0 msgr2=0x7fa580074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:04.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.685+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 shutdown_connections 2024-03-25T14:31:04.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.685+0000 7fa586eed640 1 -- 172.21.15.63:0/1925721419 wait complete. 2024-03-25T14:31:04.691 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:04.685+0000 7fa586eed640 1 librados: shutdown 2024-03-25T14:31:04.700 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:11.508 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.623+0000 7fd9be840640 10 monclient: get_monmap_and_config 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.623+0000 7fd9be840640 10 monclient: build_initial_monmap 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.623+0000 7fd9be840640 10 monclient: monmap: 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:31:11.625016+0000 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:31:11.625016+0000 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:31:11.629 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:31:11.630 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.624+0000 7fd9be840640 1 Processor -- start 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 1 -- start start 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient: init 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient(hunting): picked mon.noname-c con 0x7fd9b806adf0 addr v1:172.21.15.153:6789/0 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient(hunting): picked mon.noname-b con 0x7fd9b80aee20 addr v1:172.21.15.63:6790/0 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b80afe80 addr v1:172.21.15.63:6789/0 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b80729c0 con 0x7fd9b80afe80 2024-03-25T14:31:11.631 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b8143e60 con 0x7fd9b806adf0 2024-03-25T14:31:11.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b813db40 con 0x7fd9b80aee20 2024-03-25T14:31:11.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:11.632 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.625+0000 7fd9be840640 10 monclient(hunting): authenticate will time out at 2034.770405s 2024-03-25T14:31:11.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.633+0000 7fd9b7fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd9b806adf0 0x7fd9b8143750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:45198/0 (socket says 172.21.15.63:45198) 2024-03-25T14:31:11.638 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.633+0000 7fd9b7fff640 1 -- 172.21.15.63:0/374515668 learned_addr learned my addr 172.21.15.63:0/374515668 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:31:11.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.652+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765055402 0 0) 0x7fd9b8143e60 con 0x7fd9b806adf0 2024-03-25T14:31:11.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.652+0000 7fd9b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.652+0000 7fd9b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.652+0000 7fd9b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.657 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.652+0000 7fd9b6ffd640 10 monclient(hunting): my global_id is 4741 2024-03-25T14:31:11.658 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.652+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a0003520 con 0x7fd9b806adf0 2024-03-25T14:31:11.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.658+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4214815243 0 0) 0x7fd9a0003520 con 0x7fd9b806adf0 2024-03-25T14:31:11.664 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 >> v1:172.21.15.63:6790/0 conn(0x7fd9b80aee20 legacy=0x7fd9b813d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 >> v1:172.21.15.63:6789/0 conn(0x7fd9b80afe80 legacy=0x7fd9b814baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T14:31:11.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 1 monclient: found mon.noname-c 2024-03-25T14:31:11.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:11.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:31:11.665 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b813dcd0 con 0x7fd9b806adf0 2024-03-25T14:31:11.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 10 monclient: _finish_auth 0 2024-03-25T14:31:11.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T14:31:11.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:11.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd9a8001690 con 0x7fd9b806adf0 2024-03-25T14:31:11.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9be840640 5 monclient: authenticate success, global_id 4741 2024-03-25T14:31:11.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9be840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:11.666 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9b6ffd640 10 monclient: dump: 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:11.667 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:11.668 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.659+0000 7fd9be840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9a80017f0 con 0x7fd9b806adf0 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9b6ffd640 1 -- 172.21.15.63:0/374515668 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd9a8001c00 con 0x7fd9b806adf0 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9b6ffd640 10 monclient: dump: 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.671 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:11.672 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:11.672 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:11.672 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:11.672 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:11.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:11.672 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 4 set_mon_vals no callback set 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 10 monclient: shutdown 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 1 -- 172.21.15.63:0/374515668 >> v1:172.21.15.153:6789/0 conn(0x7fd9b806adf0 legacy=0x7fd9b8143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 1 -- 172.21.15.63:0/374515668 shutdown_connections 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 1 -- 172.21.15.63:0/374515668 >> 172.21.15.63:0/374515668 conn(0x7fd9b80b4a80 msgr2=0x7fd9b80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 1 -- 172.21.15.63:0/374515668 shutdown_connections 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.666+0000 7fd9be840640 1 -- 172.21.15.63:0/374515668 wait complete. 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.667+0000 7fd9be840640 10 monclient: build_initial_monmap 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.667+0000 7fd9be840640 10 monclient: monmap: 2024-03-25T14:31:11.674 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.667+0000 7fd9be840640 1 librados: starting msgr at 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.667+0000 7fd9be840640 1 librados: starting objecter 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.667+0000 7fd9be840640 1 Processor -- start 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 1 -- start start 2024-03-25T14:31:11.675 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 1 librados: setting wanted keys 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 1 librados: calling monclient init 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 10 monclient: init 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b806adf0 addr v1:172.21.15.63:6789/0 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 10 monclient(hunting): picked mon.noname-b con 0x7fd9b80afe80 addr v1:172.21.15.153:6789/0 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 10 monclient(hunting): picked mon.noname-c con 0x7fd9b8136350 addr v1:172.21.15.63:6790/0 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b8140870 con 0x7fd9b806adf0 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b8140b50 con 0x7fd9b80afe80 2024-03-25T14:31:11.676 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b8140e30 con 0x7fd9b8136350 2024-03-25T14:31:11.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.668+0000 7fd9be840640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:11.677 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.669+0000 7fd9be840640 10 monclient(hunting): authenticate will time out at 2034.813524s 2024-03-25T14:31:11.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.674+0000 7fd9b77fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd9b80afe80 0x7fd9b8135c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:45230/0 (socket says 172.21.15.63:45230) 2024-03-25T14:31:11.679 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.674+0000 7fd9b77fe640 1 -- 172.21.15.63:0/2923741857 learned_addr learned my addr 172.21.15.63:0/2923741857 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:31:11.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.675+0000 7fd9b4ff9640 10 client.?.objecter ms_handle_connect 0x7fd9b80afe80 2024-03-25T14:31:11.680 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.675+0000 7fd9b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:11.694 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.689+0000 7fd9b4ff9640 10 client.?.objecter ms_handle_connect 0x7fd9b8136350 2024-03-25T14:31:11.695 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.689+0000 7fd9b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:11.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.694+0000 7fd9b4ff9640 10 client.?.objecter ms_handle_connect 0x7fd9b806adf0 2024-03-25T14:31:11.699 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.694+0000 7fd9b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:11.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.700+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323422071 0 0) 0x7fd9b8140b50 con 0x7fd9b80afe80 2024-03-25T14:31:11.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.700+0000 7fd9b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.700+0000 7fd9b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.700+0000 7fd9b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.700+0000 7fd9b4ff9640 10 monclient(hunting): my global_id is 4744 2024-03-25T14:31:11.705 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.700+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a4004eb0 con 0x7fd9b80afe80 2024-03-25T14:31:11.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.709+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721766802 0 0) 0x7fd9b8140870 con 0x7fd9b806adf0 2024-03-25T14:31:11.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.709+0000 7fd9b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.709+0000 7fd9b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.709+0000 7fd9b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.709+0000 7fd9b4ff9640 10 monclient(hunting): my global_id is 4740 2024-03-25T14:31:11.715 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.709+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8140b50 con 0x7fd9b806adf0 2024-03-25T14:31:11.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.710+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782081300 0 0) 0x7fd9b8140e30 con 0x7fd9b8136350 2024-03-25T14:31:11.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.710+0000 7fd9b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T14:31:11.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.711+0000 7fd9b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:11.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.711+0000 7fd9b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:11.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.711+0000 7fd9b4ff9640 10 monclient(hunting): my global_id is 4709 2024-03-25T14:31:11.716 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.711+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8140870 con 0x7fd9b8136350 2024-03-25T14:31:11.733 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.728+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2923688429 0 0) 0x7fd9b8140870 con 0x7fd9b8136350 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.728+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9b8140e30 con 0x7fd9b8136350 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.728+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd9b0001a90 con 0x7fd9b8136350 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.728+0000 7fd9b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.728+0000 7fd9b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.728+0000 7fd9b4ff9640 10 monclient(hunting): dump: 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.734 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:11.735 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:11.735 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:11.735 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:11.735 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:11.735 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.728+0000 7fd9b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:11.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2338352936 0 0) 0x7fd9b8140e30 con 0x7fd9b8136350 2024-03-25T14:31:11.740 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 >> v1:172.21.15.153:6789/0 conn(0x7fd9b80afe80 legacy=0x7fd9b8135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 >> v1:172.21.15.63:6789/0 conn(0x7fd9b806adf0 legacy=0x7fd9b813f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 1 monclient: found mon.c 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b813f7e0 con 0x7fd9b8136350 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 10 monclient: _finish_auth 0 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 5 monclient: authenticate success, global_id 4709 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 10 monclient: _renew_subs 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:31:11.741 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b81d3940 con 0x7fd9b8136350 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 10 monclient: _renew_subs 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b81d3ec0 con 0x7fd9b8136350 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.734+0000 7fd9be840640 1 librados: init done 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.735+0000 7fd9be840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.735+0000 7fd9be840640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.735+0000 7fd9be840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:31:11.742 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.735+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b81d3080 con 0x7fd9b8136350 2024-03-25T14:31:11.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.949+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9b0001fa0 con 0x7fd9b8136350 2024-03-25T14:31:11.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.949+0000 7fd9b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:11.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.949+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7fd9b0003b10 con 0x7fd9b8136350 2024-03-25T14:31:11.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.949+0000 7fd9b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:11.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.949+0000 7fd9b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T14:31:11.954 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.949+0000 7fd9b4ff9640 10 monclient: dump: 2024-03-25T14:31:11.954 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:11.955 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:11.949+0000 7fd9b67fc640 4 set_mon_vals no callback set 2024-03-25T14:31:12.038 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.033+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7fd9b0003c70 con 0x7fd9b8136350 2024-03-25T14:31:12.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.034+0000 7fd9b7fff640 10 monclient: get_auth_request con 0x7fd9a406de50 auth_method 0 2024-03-25T14:31:12.039 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.034+0000 7fd9b4ff9640 10 client.4709.objecter ms_handle_connect 0x7fd9a406de50 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7fd9b00537d0 con 0x7fd9b8136350 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 10 client.4709.objecter ms_dispatch 0x7fd9b8000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 3 client.4709.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 3 client.4709.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9b004f110 con 0x7fd9b8136350 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:12.046 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.040+0000 7fd9b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.291+0000 7fd9be840640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.291+0000 7fd9be840640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.291+0000 7fd9be840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.63:6790/0 2024-03-25T14:31:12.297 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.291+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 --> v1:172.21.15.63:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9b806ed20 con 0x7fd9b8136350 2024-03-25T14:31:12.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.430+0000 7fd9b4ff9640 1 -- 172.21.15.63:0/2923741857 <== mon.2 v1:172.21.15.63:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7fd9b806ed20 con 0x7fd9b8136350 2024-03-25T14:31:12.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.431+0000 7fd9b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:12.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.431+0000 7fd9b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:12.486 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:31:12.486 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:12.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 10 librados: watch_flush enter 2024-03-25T14:31:12.487 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 10 librados: watch_flush exit 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 10 client.4709.objecter successfully canceled tick 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 >> v1:172.21.15.153:6800/1688255552 conn(0x7fd9a406de50 legacy=0x7fd9a4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 10 monclient: shutdown 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 >> v1:172.21.15.63:6790/0 conn(0x7fd9b8136350 legacy=0x7fd9b81d3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 shutdown_connections 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 >> 172.21.15.63:0/2923741857 conn(0x7fd9b80b4a80 msgr2=0x7fd9b8139110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 shutdown_connections 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 1 -- 172.21.15.63:0/2923741857 wait complete. 2024-03-25T14:31:12.488 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:12.433+0000 7fd9be840640 1 librados: shutdown 2024-03-25T14:31:12.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:13.170 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:51.682 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.214 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:51.673 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.215 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:51.674 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.240 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:51.752 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.260 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:51.718 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.279 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:51.790 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.280 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:11:51.792 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.284 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:51.796 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:13.318 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:11:51.830 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:19.296 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T14:31:19.397 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.392+0000 7f8f2b065640 10 monclient: get_monmap_and_config 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.392+0000 7f8f2b065640 10 monclient: build_initial_monmap 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.392+0000 7f8f2b065640 10 monclient: monmap: 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:31:19.393772+0000 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:31:19.393772+0000 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:19.398 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-b 2024-03-25T14:31:19.399 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:19.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.393+0000 7f8f2b065640 1 Processor -- start 2024-03-25T14:31:19.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.393+0000 7f8f2b065640 1 -- start start 2024-03-25T14:31:19.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.393+0000 7f8f2b065640 10 monclient: init 2024-03-25T14:31:19.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:19.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T14:31:19.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 10 monclient(hunting): picked mon.noname-a con 0x7f8f240a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:31:19.400 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 10 monclient(hunting): picked mon.noname-c con 0x7f8f2413f710 addr v1:172.21.15.153:6789/0 2024-03-25T14:31:19.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 10 monclient(hunting): picked mon.noname-b con 0x7f8f241430c0 addr v1:172.21.15.63:6790/0 2024-03-25T14:31:19.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f24142520 con 0x7f8f240a98f0 2024-03-25T14:31:19.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240b28a0 con 0x7f8f2413f710 2024-03-25T14:31:19.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240b2a30 con 0x7f8f241430c0 2024-03-25T14:31:19.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:19.401 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.394+0000 7f8f2b065640 10 monclient(hunting): authenticate will time out at 2042.539297s 2024-03-25T14:31:19.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.399+0000 7f8f23fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8f2413f710 0x7f8f241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.63:49926/0 (socket says 172.21.15.63:49926) 2024-03-25T14:31:19.404 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.399+0000 7f8f23fff640 1 -- 172.21.15.63:0/2750238609 learned_addr learned my addr 172.21.15.63:0/2750238609 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:31:19.411 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.406+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435430346 0 0) 0x7f8f240b28a0 con 0x7f8f2413f710 2024-03-25T14:31:19.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.407+0000 7f8f237fe640 10 monclient(hunting): _negotiate 2024-03-25T14:31:19.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.407+0000 7f8f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:19.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.407+0000 7f8f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:19.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.407+0000 7f8f237fe640 10 monclient(hunting): my global_id is 4747 2024-03-25T14:31:19.412 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.407+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f14003520 con 0x7f8f2413f710 2024-03-25T14:31:19.419 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209607194 0 0) 0x7f8f14003520 con 0x7f8f2413f710 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 >> v1:172.21.15.63:6790/0 conn(0x7f8f241430c0 legacy=0x7f8f2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 >> v1:172.21.15.63:6789/0 conn(0x7f8f240a98f0 legacy=0x7f8f240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 10 monclient: _finish_hunting 0 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 1 monclient: found mon.noname-c 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.153:6789/0 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f240b44c0 con 0x7f8f2413f710 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 10 monclient: _finish_auth 0 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 10 monclient: _check_auth_tickets 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f10001690 con 0x7f8f2413f710 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:19.420 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f237fe640 10 monclient: dump: 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f2b065640 5 monclient: authenticate success, global_id 4747 2024-03-25T14:31:19.421 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.414+0000 7f8f2b065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T14:31:19.431 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 <== mon.1 v1:172.21.15.153:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f10001860 con 0x7f8f2413f710 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f237fe640 1 -- 172.21.15.63:0/2750238609 <== mon.1 v1:172.21.15.153:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f10001cb0 con 0x7f8f2413f710 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f2b065640 10 monclient: get_monmap_and_config success 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f2b065640 4 set_mon_vals no callback set 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.426+0000 7f8f237fe640 10 monclient: dump: 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.432 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:19.433 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:19.433 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:19.433 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:19.433 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:19.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.427+0000 7f8f2b065640 10 monclient: shutdown 2024-03-25T14:31:19.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.427+0000 7f8f2b065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:19.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.427+0000 7f8f2b065640 1 -- 172.21.15.63:0/2750238609 >> v1:172.21.15.153:6789/0 conn(0x7f8f2413f710 legacy=0x7f8f241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.427+0000 7f8f2b065640 1 -- 172.21.15.63:0/2750238609 shutdown_connections 2024-03-25T14:31:19.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.427+0000 7f8f2b065640 1 -- 172.21.15.63:0/2750238609 >> 172.21.15.63:0/2750238609 conn(0x7f8f240b7ea0 msgr2=0x7f8f240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:19.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.427+0000 7f8f2b065640 1 -- 172.21.15.63:0/2750238609 shutdown_connections 2024-03-25T14:31:19.435 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.427+0000 7f8f2b065640 1 -- 172.21.15.63:0/2750238609 wait complete. 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 10 monclient: build_initial_monmap 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 10 monclient: monmap: 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.noname-b 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.noname-c 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:19.436 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 1 librados: starting msgr at 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 1 librados: starting objecter 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 1 Processor -- start 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 1 -- start start 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 1 librados: setting wanted keys 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 1 librados: calling monclient init 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.428+0000 7f8f2b065640 10 monclient: init 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 10 monclient: _reopen_session rank -1 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 10 monclient(hunting): picked mon.noname-a con 0x7f8f240a98f0 addr v1:172.21.15.63:6789/0 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 10 monclient(hunting): picked mon.noname-c con 0x7f8f2413f710 addr v1:172.21.15.63:6790/0 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 10 monclient(hunting): picked mon.noname-b con 0x7f8f241430c0 addr v1:172.21.15.153:6789/0 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f24136760 con 0x7f8f240a98f0 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f241368f0 con 0x7f8f241430c0 2024-03-25T14:31:19.437 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 1 -- --> v1:172.21.15.63:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2413b410 con 0x7f8f2413f710 2024-03-25T14:31:19.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 10 monclient(hunting): _renew_subs 2024-03-25T14:31:19.438 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.429+0000 7f8f2b065640 10 monclient(hunting): authenticate will time out at 2042.574264s 2024-03-25T14:31:19.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.443+0000 7f8f23fff640 1 --1- >> v1:172.21.15.63:6790/0 conn(0x7f8f2413f710 0x7f8f241374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.63:6790/0 says I am v1:172.21.15.63:41010/0 (socket says 172.21.15.63:41010) 2024-03-25T14:31:19.448 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.443+0000 7f8f23fff640 1 -- 172.21.15.63:0/4246493772 learned_addr learned my addr 172.21.15.63:0/4246493772 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-03-25T14:31:19.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.445+0000 7f8f217fa640 10 client.?.objecter ms_handle_connect 0x7f8f2413f710 2024-03-25T14:31:19.450 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.445+0000 7f8f217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:19.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.449+0000 7f8f217fa640 10 client.?.objecter ms_handle_connect 0x7f8f241430c0 2024-03-25T14:31:19.454 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.449+0000 7f8f217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:19.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.457+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.2 v1:172.21.15.63:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154650052 0 0) 0x7f8f2413b410 con 0x7f8f2413f710 2024-03-25T14:31:19.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.457+0000 7f8f217fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:19.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.457+0000 7f8f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:19.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.457+0000 7f8f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:19.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.457+0000 7f8f217fa640 10 monclient(hunting): my global_id is 4715 2024-03-25T14:31:19.462 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.457+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.63:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f08004eb0 con 0x7f8f2413f710 2024-03-25T14:31:19.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.458+0000 7f8f217fa640 10 client.?.objecter ms_handle_connect 0x7f8f240a98f0 2024-03-25T14:31:19.463 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.458+0000 7f8f217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T14:31:19.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.461+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746603048 0 0) 0x7f8f241368f0 con 0x7f8f241430c0 2024-03-25T14:31:19.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.461+0000 7f8f217fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:19.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.461+0000 7f8f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:19.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.461+0000 7f8f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:19.466 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.461+0000 7f8f217fa640 10 monclient(hunting): my global_id is 4750 2024-03-25T14:31:19.467 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.461+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f2413b410 con 0x7f8f241430c0 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.473+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669921268 0 0) 0x7f8f24136760 con 0x7f8f240a98f0 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.473+0000 7f8f217fa640 10 monclient(hunting): _negotiate 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.473+0000 7f8f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T14:31:19.478 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.473+0000 7f8f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T14:31:19.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.473+0000 7f8f217fa640 10 monclient(hunting): my global_id is 4743 2024-03-25T14:31:19.479 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.473+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f241368f0 con 0x7f8f240a98f0 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.476+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.2 v1:172.21.15.63:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326748900 0 0) 0x7f8f08004eb0 con 0x7f8f2413f710 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.476+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.63:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f24136760 con 0x7f8f2413f710 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.476+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.2 v1:172.21.15.63:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f10001b70 con 0x7f8f2413f710 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.476+0000 7f8f217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.476+0000 7f8f217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.476+0000 7f8f217fa640 10 monclient(hunting): dump: 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:19.481 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:19.482 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.476+0000 7f8f217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T14:31:19.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.479+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 704830327 0 0) 0x7f8f2413b410 con 0x7f8f241430c0 2024-03-25T14:31:19.484 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.479+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f08004eb0 con 0x7f8f241430c0 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.479+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f1c001b10 con 0x7f8f241430c0 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.479+0000 7f8f217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.479+0000 7f8f217fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.479+0000 7f8f217fa640 10 monclient(hunting): dump: 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:19.485 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:19.486 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:19.486 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:19.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 71495429 0 0) 0x7f8f08004eb0 con 0x7f8f241430c0 2024-03-25T14:31:19.489 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 >> v1:172.21.15.63:6790/0 conn(0x7f8f2413f710 legacy=0x7f8f241374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 >> v1:172.21.15.63:6789/0 conn(0x7f8f240a98f0 legacy=0x7f8f24139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 10 monclient: _finish_hunting 0 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 1 monclient: found mon.b 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2413b5a0 con 0x7f8f241430c0 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 10 monclient: _finish_auth 0 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 10 monclient: _check_auth_tickets 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 5 monclient: authenticate success, global_id 4750 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 10 monclient: _renew_subs 2024-03-25T14:31:19.490 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:31:19.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f241da8f0 con 0x7f8f241430c0 2024-03-25T14:31:19.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 10 client.4750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T14:31:19.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 10 monclient: _renew_subs 2024-03-25T14:31:19.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:31:19.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f241e3550 con 0x7f8f241430c0 2024-03-25T14:31:19.491 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.484+0000 7f8f2b065640 1 librados: init done 2024-03-25T14:31:19.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.485+0000 7f8f2b065640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:19.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.485+0000 7f8f2b065640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:19.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.485+0000 7f8f2b065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:31:19.492 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.485+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f240b3ef0 con 0x7f8f241430c0 2024-03-25T14:31:19.764 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.759+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f1c002000 con 0x7f8f241430c0 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.759+0000 7f8f217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.759+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1859937699 0 0) 0x7f8f1c003b30 con 0x7f8f241430c0 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.759+0000 7f8f217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.759+0000 7f8f217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.759+0000 7f8f217fa640 10 monclient: dump: 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:fsid d1a7f567-bb16-4bed-8dc7-9f0ba166adec 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:created 2024-03-25T14:18:48.449049+0000 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-03-25T14:31:19.765 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 3 2024-03-25T14:31:19.766 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.a 2024-03-25T14:31:19.766 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.153:6789/0 mon.b 2024-03-25T14:31:19.766 INFO:teuthology.misc.health.smithi063.stderr:2: v1:172.21.15.63:6790/0 mon.c 2024-03-25T14:31:19.766 INFO:teuthology.misc.health.smithi063.stderr: 2024-03-25T14:31:19.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.759+0000 7f8f22ffd640 4 set_mon_vals no callback set 2024-03-25T14:31:19.766 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.760+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1480125698 0 0) 0x7f8f1c019b00 con 0x7f8f241430c0 2024-03-25T14:31:19.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.760+0000 7f8f28e01640 10 monclient: get_auth_request con 0x7f8f0806df60 auth_method 0 2024-03-25T14:31:19.767 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.760+0000 7f8f217fa640 10 client.4750.objecter ms_handle_connect 0x7f8f0806df60 2024-03-25T14:31:19.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3680088250 0 0) 0x7f8f1c053830 con 0x7f8f241430c0 2024-03-25T14:31:19.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 10 client.4750.objecter ms_dispatch 0x7f8f24000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T14:31:19.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 3 client.4750.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T14:31:19.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 3 client.4750.objecter handle_osd_map decoding full epoch 12 2024-03-25T14:31:19.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 20 client.4750.objecter dump_active .. 0 homeless 2024-03-25T14:31:19.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f1c04f170 con 0x7f8f241430c0 2024-03-25T14:31:19.770 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T14:31:19.771 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:19.765+0000 7f8f217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T14:31:20.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.020+0000 7f8f2b065640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:20.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.020+0000 7f8f2b065640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:20.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.020+0000 7f8f2b065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.153:6789/0 2024-03-25T14:31:20.026 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.020+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f24139960 con 0x7f8f241430c0 2024-03-25T14:31:20.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.046+0000 7f8f217fa640 1 -- 172.21.15.63:0/4246493772 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2044434634) 0x7f8f1c051270 con 0x7f8f241430c0 2024-03-25T14:31:20.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.046+0000 7f8f217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T14:31:20.051 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.046+0000 7f8f217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T14:31:20.051 INFO:teuthology.misc.health.smithi063.stdout: 2024-03-25T14:31:20.051 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"d1a7f567-bb16-4bed-8dc7-9f0ba166adec","created":"2024-03-25T14:19:49.151843+0000","modified":"2024-03-25T14:29:57.858425+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T14:29:57.858425+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"40c78739-8253-4432-bca1-f84c1a0e50a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"06668c03-387c-4b0b-8489-9e3661b8144f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8962c36e-fad1-4188-8093-f42c888c9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"07600efd-4d69-477c-a480-b036389aeb1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3b31d7df-e7c2-4d5b-97b1-ae422be070d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"535c7ab0-1e04-488b-9b5b-c04f91b6ca9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"65bad5c1-0847-41d4-b4de-e5c8572569e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce24bc86-436e-44aa-8020-5fa5e0250e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T14:31:20.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 10 librados: watch_flush enter 2024-03-25T14:31:20.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 10 librados: watch_flush exit 2024-03-25T14:31:20.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 20 client.4750.objecter shutdown clearing up homeless session... 2024-03-25T14:31:20.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 10 client.4750.objecter successfully canceled tick 2024-03-25T14:31:20.054 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 >> v1:172.21.15.153:6800/1688255552 conn(0x7f8f0806df60 legacy=0x7f8f08070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 10 monclient: shutdown 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.048+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 >> v1:172.21.15.153:6789/0 conn(0x7f8f241430c0 legacy=0x7f8f241e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.049+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 shutdown_connections 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.049+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 >> 172.21.15.63:0/4246493772 conn(0x7f8f240b7ea0 msgr2=0x7f8f2413c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.049+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 shutdown_connections 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.049+0000 7f8f2b065640 1 -- 172.21.15.63:0/4246493772 wait complete. 2024-03-25T14:31:20.055 INFO:teuthology.misc.health.smithi063.stderr:2024-03-25T14:31:20.049+0000 7f8f2b065640 1 librados: shutdown 2024-03-25T14:31:20.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T14:31:20.067 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T14:31:20.069 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T14:31:20.070 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T14:31:20.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:31:22.131 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:11:26.735 35371== 2024-03-25T14:31:22.131 INFO:tasks.ceph.osd.0.smithi063.stderr:==00:00:11:26.735 35371== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:31:22.170 INFO:tasks.ceph.osd.0.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:31:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:31:26.172 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T14:31:26.173 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T14:31:26.173 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T14:31:26.173 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:31:28.234 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:11:32.847 35370== 2024-03-25T14:31:28.234 INFO:tasks.ceph.osd.1.smithi063.stderr:==00:00:11:32.847 35370== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:31:28.271 INFO:tasks.ceph.osd.1.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:31:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:31:32.276 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T14:31:32.277 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T14:31:32.277 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T14:31:32.277 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:31:33.282 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:11.794 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.288 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:11.800 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.321 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:11.778 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.328 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:11.840 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.329 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:11.788 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.331 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:11.843 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.338 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:11.850 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.350 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:11.809 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:33.364 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:11.876 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:34.151 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:11:38.742 35372== 2024-03-25T14:31:34.151 INFO:tasks.ceph.osd.2.smithi063.stderr:==00:00:11:38.742 35372== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:31:34.193 INFO:tasks.ceph.osd.2.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:31:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:31:38.380 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T14:31:38.380 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T14:31:38.380 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T14:31:38.380 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:31:40.308 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:11:44.899 35373== 2024-03-25T14:31:40.308 INFO:tasks.ceph.osd.3.smithi063.stderr:==00:00:11:44.899 35373== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:31:40.343 INFO:tasks.ceph.osd.3.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:31:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:31:44.483 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T14:31:44.484 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T14:31:44.484 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T14:31:44.484 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:31:46.460 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:11:51.073 35381== 2024-03-25T14:31:46.460 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:11:51.073 35381== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:31:46.497 INFO:tasks.ceph.osd.4.smithi153.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:31:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:31:50.586 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T14:31:50.587 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T14:31:50.587 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T14:31:50.587 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:31:52.401 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:11:57.014 35382== 2024-03-25T14:31:52.401 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:11:57.014 35382== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:31:52.433 INFO:tasks.ceph.osd.5.smithi153.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:31:53.430 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:31.941 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.433 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:31.945 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.457 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:31.969 35056== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.476 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:31.935 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.482 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:31.993 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.483 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:31.942 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.487 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:31.999 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.497 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:31.956 34829== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:53.504 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:32.016 35059== Warning: unimplemented fcntl command: 1036 2024-03-25T14:31:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:31:56.689 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T14:31:56.690 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T14:31:56.691 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T14:31:56.692 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:31:58.392 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:12:03.003 35383== 2024-03-25T14:31:58.392 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:12:03.003 35383== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:31:58.439 INFO:tasks.ceph.osd.6.smithi153.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:32:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:32:02.794 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T14:32:02.795 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T14:32:02.795 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T14:32:02.795 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:32:04.217 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:12:08.822 35384== 2024-03-25T14:32:04.217 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:12:08.822 35384== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:32:04.247 INFO:tasks.ceph.osd.7.smithi153.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:32:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:32:08.897 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T14:32:08.898 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T14:32:08.898 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T14:32:08.899 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T14:32:08.899 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:32:08.926 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T14:32:08.927 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T14:32:08.927 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:32:08.928 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:32:08.922+0000 7f3dfd4b0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34827) UID: 0 2024-03-25T14:32:08.929 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T14:32:08.922+0000 7f3dfd4b0640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T14:32:08.955 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T14:32:08.956 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T14:32:08.956 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T14:32:08.956 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:32:08.985 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-03-25T14:32:08.968+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35025) UID: 0 2024-03-25T14:32:08.986 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-03-25T14:32:08.970+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T14:32:09.552 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:48.063 35056== 2024-03-25T14:32:09.552 INFO:tasks.ceph.mon.a.smithi063.stderr:==00:00:12:48.063 35056== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:32:09.561 INFO:tasks.ceph.mon.a.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:32:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:32:15.058 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T14:32:15.059 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T14:32:15.059 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T14:32:15.059 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:32:15.077 INFO:tasks.ceph.mon.c.smithi063.stderr:2024-03-25T14:32:15.058+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35027) UID: 0 2024-03-25T14:32:15.078 INFO:tasks.ceph.mon.c.smithi063.stderr:2024-03-25T14:32:15.060+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T14:32:15.722 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:54.233 35059== 2024-03-25T14:32:15.722 INFO:tasks.ceph.mon.c.smithi063.stderr:==00:00:12:54.233 35059== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:32:15.732 INFO:tasks.ceph.mon.c.smithi063.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:32:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:32:21.162 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T14:32:21.162 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T14:32:21.163 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T14:32:21.163 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T14:32:21.179 INFO:tasks.ceph.mon.b.smithi153.stderr:2024-03-25T14:32:21.162+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34825) UID: 0 2024-03-25T14:32:21.179 INFO:tasks.ceph.mon.b.smithi153.stderr:2024-03-25T14:32:21.164+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T14:33:22.509 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:00.966 34829== 2024-03-25T14:33:22.509 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:00.966 34829== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T14:33:22.517 INFO:tasks.ceph.mon.b.smithi153.stderr:daemon-helper: command failed with exit status 42 2024-03-25T14:33:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T14:33:28.278 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T14:33:28.279 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T14:33:28.279 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T14:33:28.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T14:33:28.324 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:33:28.324 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T14:33:28.391 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:33:28.391 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T14:33:28.443 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:33:28.443 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T14:33:28.500 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:33:28.500 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T14:33:28.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:33:28.550 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T14:33:28.650 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:33:28.650 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T14:33:28.715 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:33:28.715 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T14:33:28.807 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:33:28.808 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T14:33:28.864 INFO:tasks.ceph:Archiving mon data... 2024-03-25T14:33:28.866 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/data/mon.a.tgz 2024-03-25T14:33:28.866 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-25T14:33:28.882 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.C02nMx4HxI 2024-03-25T14:33:28.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.C02nMx4HxI 2024-03-25T14:33:28.987 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.C02nMx4HxI 2024-03-25T14:33:29.073 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.C02nMx4HxI is 136KB 2024-03-25T14:33:29.129 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.C02nMx4HxI 2024-03-25T14:33:29.144 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/data/mon.c.tgz 2024-03-25T14:33:29.144 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-03-25T14:33:29.197 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.N6ENOzZkIH 2024-03-25T14:33:29.197 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.N6ENOzZkIH 2024-03-25T14:33:29.285 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.N6ENOzZkIH 2024-03-25T14:33:29.369 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.N6ENOzZkIH is 136KB 2024-03-25T14:33:29.420 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.N6ENOzZkIH 2024-03-25T14:33:29.434 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/data/mon.b.tgz 2024-03-25T14:33:29.434 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-25T14:33:29.451 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.envwwFcNY8 2024-03-25T14:33:29.451 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.envwwFcNY8 2024-03-25T14:33:29.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.envwwFcNY8 2024-03-25T14:33:29.645 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.envwwFcNY8 is 136KB 2024-03-25T14:33:29.671 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.envwwFcNY8 2024-03-25T14:33:29.685 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T14:33:29.686 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T14:33:29.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T14:33:30.073 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T14:33:30.347 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T14:33:30.349 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/remote/smithi063/crash 2024-03-25T14:33:30.350 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T14:33:30.377 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/remote/smithi153/crash 2024-03-25T14:33:30.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T14:33:30.404 INFO:tasks.ceph:Compressing logs... 2024-03-25T14:33:30.404 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T14:33:30.422 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T14:33:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-03-25T14:33:30.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T14:33:30.491 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T14:33:30.491 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35023.log 2024-03-25T14:33:30.491 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T14:33:30.491 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35023.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T14:33:30.491 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T14:33:30.491 INFO:teuthology.orchestra.run.smithi153.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T14:33:30.491 INFO:teuthology.orchestra.run.smithi153.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35023.log.gz 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose --/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/mon.b.log 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/mon.b.log: 71.3% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35274.log 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35185.log 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35063.log 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35185.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35274.log.gz 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35063.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T14:33:30.492 INFO:teuthology.orchestra.run.smithi153.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35185.log.gz 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35063.log.gz 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35103.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35143.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35103.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35234.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35103.log.gz 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mgr.x.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35143.log.gz 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35234.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.35234.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.34983.log 2024-03-25T14:33:30.493 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:33:30.494 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T14:33:30.497 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34983.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34983.log.gz 2024-03-25T14:33:30.502 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41803.log 2024-03-25T14:33:30.502 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35538.log 2024-03-25T14:33:30.502 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-03-25T14:33:30.502 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2024-03-25T14:33:30.502 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40048.log 2024-03-25T14:33:30.502 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35538.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41803.log.gz 2024-03-25T14:33:30.502 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38914.log 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35538.log.gz 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.38914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39267.log 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40048.log.gz 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38914.log.gz 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39267.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39853.log 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39267.log.gz 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41062.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T14:33:30.503 INFO:teuthology.orchestra.run.smithi063.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39853.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39853.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39619.log 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T14:33:30.504 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41882.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr: 73.6% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39502.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39619.log.gz 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41882.log.gz 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40945.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39502.log: /var/log/ceph/ceph-client.admin.40906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42002.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39502.log.gz 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40633.log 2024-03-25T14:33:30.505 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40945.log.gz 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40087.log 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42002.log.gz 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40633.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz -5 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41963.log 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40087.log: gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40633.log.gz -5 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40282.log 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40867.log 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40087.log.gz 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40282.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41963.log.gz 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39580.log 2024-03-25T14:33:30.506 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40282.log.gz 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41569.log 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40867.log.gz 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39580.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41569.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39580.log.gz 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.38994.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41647.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41569.log.gz 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42119.log 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.42119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-03-25T14:33:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41491.log 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42119.log.gz 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39658.log 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40360.log 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37343.log 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40204.log 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39658.log.gz 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.37343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40360.log.gz 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40516.log 2024-03-25T14:33:30.508 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37343.log.gz 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40204.log.gz 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40516.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40984.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40516.log.gz 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39228.log 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41179.log 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39228.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35369.log 2024-03-25T14:33:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40672.log 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39228.log.gz 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41218.log 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35369.log.gz 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41179.log.gz 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40672.log: /var/log/ceph/ceph-client.admin.41218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40243.log 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40672.log.gz 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40828.log 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41218.log.gz 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.40243.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-03-25T14:33:30.510 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40438.log 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40243.log.gz 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.38953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40828.log.gz 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40438.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40438.log.gz 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35067.log 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.35067.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40594.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35067.log.gz 2024-03-25T14:33:30.511 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz -5 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40165.log 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5/var/log/ceph/ceph-client.admin.39423.log: --verbose 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr: -- /var/log/ceph/ceph-client.admin.40711.log 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40165.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29201.log 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40165.log.gz 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41924.log 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39189.log 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.tmp-client.admin.29201.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29201.log.gz 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40711.log.gz 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41608.log 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40321.log 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41924.log.gz 2024-03-25T14:33:30.512 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41608.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39189.log.gz 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39150.log 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40321.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41608.log.gz 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39150.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40321.log.gz 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41842.log 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39150.log.gz 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40009.log 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41101.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41842.log.gz 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35396.log 2024-03-25T14:33:30.513 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40009.log: /var/log/ceph/ceph-client.admin.35396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40555.log 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40009.log.gz/var/log/ceph/ceph-client.admin.41257.log: 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr: 85.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.35396.log.gz -5 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41335.log 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40126.log 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40555.log: gzip -5/var/log/ceph/ceph-client.admin.41335.log: --verbose -- /var/log/ceph/ceph-client.admin.39931.log 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40126.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40555.log.gz 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41335.log.gz 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40126.log.gz 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2024-03-25T14:33:30.514 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39931.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40750.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0%/var/log/ceph/ceph-client.admin.41530.log: -- replaced with /var/log/ceph/ceph-client.admin.39931.log.gz 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42158.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.40750.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42158.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39970.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.2.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42158.log.gz 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39970.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39970.log.gz 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39892.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39814.log 2024-03-25T14:33:30.515 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.41725.log: /var/log/ceph/ceph-client.admin.39892.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.39892.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38684.log 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36418.log 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.38684.log: /var/log/ceph/ceph-client.admin.39814.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39814.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38684.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39775.log 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39697.log 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.36418.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39775.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36418.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39775.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39736.log 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39033.log 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39736.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39697.log.gz 2024-03-25T14:33:30.516 INFO:teuthology.orchestra.run.smithi063.stderr: -- /var/log/ceph/ceph-client.admin.42041.log 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39736.log.gz 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42080.log 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39033.log: /var/log/ceph/ceph-client.admin.42041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42197.log 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42080.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39033.log.gz 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42041.log.gz 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42080.log.gz 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.42197.log: /var/log/ceph/ceph-client.admin.41374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39541.log 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mon.c.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42197.log.gz 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-03-25T14:33:30.517 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.39541.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39541.log.gz 2024-03-25T14:33:30.555 INFO:teuthology.orchestra.run.smithi153.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T14:33:31.287 INFO:teuthology.orchestra.run.smithi063.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T14:33:31.351 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T14:33:31.372 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T14:33:31.402 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T14:33:31.546 INFO:teuthology.orchestra.run.smithi063.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T14:33:31.604 INFO:teuthology.orchestra.run.smithi063.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T14:33:31.636 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T14:33:31.687 INFO:teuthology.orchestra.run.smithi063.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T14:33:35.602 INFO:teuthology.orchestra.run.smithi063.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T14:33:35.684 INFO:teuthology.orchestra.run.smithi153.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T14:33:35.686 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T14:33:35.686 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m5.226s 2024-03-25T14:33:35.686 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m8.848s 2024-03-25T14:33:35.686 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.325s 2024-03-25T14:33:35.688 INFO:teuthology.orchestra.run.smithi063.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T14:33:35.690 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-25T14:33:35.690 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m5.256s 2024-03-25T14:33:35.690 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m14.220s 2024-03-25T14:33:35.690 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.515s 2024-03-25T14:33:35.691 INFO:tasks.ceph:Archiving logs... 2024-03-25T14:33:35.691 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/remote/smithi063/log 2024-03-25T14:33:35.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T14:33:38.217 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/remote/smithi153/log 2024-03-25T14:33:38.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T14:33:39.823 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T14:33:42.340 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0c958e3b547047fca149253c0083cdb4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T14:33:42.344 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T14:33:42.353 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T14:33:42.381 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T14:33:42.382 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T14:33:42.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T14:33:42.480 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T14:33:42.488 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T14:33:42.489 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:33:42.491 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T14:33:42.505 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:33:42.508 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-25T14:33:42.509 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:33:42.509 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-25T14:33:42.509 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +810us[ +810us] +/- 80ms 2024-03-25T14:33:42.509 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -1654us[-1611us] +/- 67ms 2024-03-25T14:33:42.510 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +1176us[+1176us] +/- 82ms 2024-03-25T14:33:42.510 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:33:42.512 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T14:33:42.512 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-25T14:33:42.512 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +831us[ +825us] +/- 80ms 2024-03-25T14:33:42.513 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -1625us[-1631us] +/- 67ms 2024-03-25T14:33:42.513 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +1189us[+1183us] +/- 82ms 2024-03-25T14:33:42.513 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T14:33:42.514 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T14:33:42.522 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T14:33:42.523 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T14:33:42.557 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T14:33:42.628 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T14:33:42.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:33:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:33:42.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T14:33:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:33:42.763 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T14:33:42.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T14:33:42.816 INFO:teuthology.task.internal:Duration was 1855.406202 seconds 2024-03-25T14:33:42.816 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T14:33:42.850 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T14:33:42.850 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T14:33:42.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T14:33:42.895 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:33:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T14:33:43.164 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T14:33:43.165 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-03-25T14:33:43.165 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T14:33:43.185 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T14:33:43.186 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T14:33:43.213 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T14:33:43.214 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T14:33:43.228 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T14:33:43.282 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T14:33:43.282 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T14:33:43.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T14:33:43.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T14:33:43.432 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T14:33:43.433 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T14:33:43.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T14:33:43.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T14:33:43.497 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:33:43.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:33:43.525 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-03-25T14:33:43.553 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-25T14:33:43.575 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:33:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:33:43.616 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T14:33:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T14:33:43.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T14:33:43.641 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T14:33:43.642 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/remote/smithi063 2024-03-25T14:33:43.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T14:33:43.692 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266/remote/smithi153 2024-03-25T14:33:43.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T14:33:43.728 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T14:33:43.729 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T14:33:43.731 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T14:33:43.777 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T14:33:43.789 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T14:33:43.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T14:33:43.818 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T14:33:43.818 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T14:33:43.821 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T14:33:43.832 INFO:teuthology.orchestra.run.smithi063.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 14:33 /home/ubuntu/cephtest 2024-03-25T14:33:43.835 INFO:teuthology.orchestra.run.smithi153.stdout: 265408 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 14:33 /home/ubuntu/cephtest 2024-03-25T14:33:43.836 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T14:33:43.851 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T14:33:43.918 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T14:33:43.945 DEBUG:teuthology.nuke:shortname: smithi063 2024-03-25T14:33:43.945 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T14:33:43.971 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-25T14:33:43.971 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T14:33:43.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:54:30.875764', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB8uAacY53DXOKDITja8/1iob9sZhkTWetwQMGMhL6oONG+hzEmsexaheGDGSltZaL6A1lH72N2TmBFw8y5A9x0='} 2024-03-25T14:33:43.997 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621266', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 13:54:30.877208', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOtg8u7uZrI0hoPbbZj5mH/sRNnx6SWjde0GamNdoFafJf5+bKIYtjqXT3MG6SDQuE/1xZlt4pWKIyEHtov3+RI='} 2024-03-25T14:33:44.029 INFO:teuthology.orchestra.console.smithi063:Power off 2024-03-25T14:33:44.029 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:33:44.052 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-03-25T14:33:44.053 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:33:44.070 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-25T14:33:44.070 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:33:44.080 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-25T14:33:44.093 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-25T14:33:44.094 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:33:44.117 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-25T14:33:48.081 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:33:48.118 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:33:48.193 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-25T14:33:48.229 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-25T14:33:52.198 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:33:52.230 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:33:52.313 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-03-25T14:33:52.341 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-25T14:33:52.414 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-03-25T14:33:52.442 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-25T14:33:52.835 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 1855.406201839447 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0c958e3b547047fca149253c0083cdb4 status: fail success: false 2024-03-25T14:33:52.835 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T14:33:52.920 INFO:teuthology.run:FAIL